Repository: getspams/spams-python (language: python)

Issue:
Permission denied: '/usr/local/tests'
Hi guys,
while installing SPAMS (`pip install spams`, I get the following error:

> ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/usr/local/tests'
> Consider using the `--user` option or check the permissions.

I know I can use the PIP option `--user`, but this happens only for SPAMS and, what is really weird, is that if I run twice `pip install spams`, I get the error only the first time, the second one no error shows up.

I am not very familiar with the build system; any clue?
Permission denied: '/usr/local/tests'
Hi guys,
while installing SPAMS (`pip install spams`, I get the following error:

> ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/usr/local/tests'
> Consider using the `--user` option or check the permissions.

I know I can use the PIP option `--user`, but this happens only for SPAMS and, what is really weird, is that if I run twice `pip install spams`, I get the error only the first time, the second one no error shows up.

I am not very familiar with the build system; any clue?


Discussion / hints:
Might be caused by the setup.py trying to install the doc, I remember I axes that part. Since the manifest already puts them in the zip, I guess that part could be deleted, since absolutely no one will go look for the doc inside system folders (and it's in the zip, it just will not get installed).
I'll look into the doc setup (and remove the copy at install which is indeed useless).
The doc issue should be fixed by #14 

@daducci could you tell me if `pip install git+https://github.com/getspams/spams-python.git@remove_doc_copy` works for you?
No, it doesn't, I still get the error:
```
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/usr/local/tests'
Consider using the `--user` option or check the permissions.
```
**Note:** not fixed by #14 

@daducci could you give more details on the environment (OS, Python version, etc.) on which you are encountering this issue ?

And if you try installing previous version from PyPI like `pip install spams==2.6.3.3` ([here](https://pypi.org/project/spams/#history) is the history of version available on PyPI), all fail or some can be installed ?

Thanks
I'm on the new Macbook M1 with OSX 12.2.1 (Monterey) and python 3.9.10 (installed via `howebrew`), and this error happens with all `spams` versions. Whats is weird is that the first time I run `pip install spams` I get this error (and, in turn, our `dmri-amico` package is not installed), whereas if I run `pip install spams` a second time, the installation runs fine and also `dmri-amico` is properly installed without any error. Weird and I don't know why this happens...
Do you have an example of a package that you can install without any issue (beside `spams` or `dmri-amico`)?
I install all other packages, e.g. `numpy`, `scipy`, `nibabel`, `networkx` etc, via `pip install <name>` (i.e., without flag `--user`) and they are installed properly. May this be due to a different settings which installs the `tests` in a wrong folder?
Is `pip install spams-bin` working ?
Yes, indeed it does, without that error! :-)
Ok, I think I understand the problem. Package with pre-built wheels do not cause any problem, but when you need some local build from source, it causes the issue.
any modern local install should try to make wheels and install that, so I'd still point the culprit to the rest of the doc and test folder from the setup.py part. I'd guess the setup tries to install the files somehow (just like with the data folder) for important stuff, while the manifest only puts them in the source zip and discards everything from the real build.

Assuming nobody would ever read or run tests from the installed files and that we only want them in the zip (since people can easily refer to those instead), then this would be doable by removing all of those references from the setup.py and only using the manifest.in file. More details here https://packaging.python.org/en/latest/guides/using-manifest-in/

If you want to put here the wheel files before install (probably under /dist in your local folder), they likely contains references to those doc and tests files.
@samuelstjean I agree. I will add all the non-standard include to the `MANIFEST.in` and remove them from the `setup.py`.
@gdurif I'm pretty sure the problem is caused by the `data_files` keyword in `setup.py` trying to install the test scripts in a system folder (`/usr/local/`). `data_files` keyword is deprecated and does not work with wheels ([see here](https://setuptools.pypa.io/en/latest/references/keywords.html)), indeed `spams-bin` works because does not use it in it's `setup.py` file.

Remove the `data_files` section from `setup.py` should solve the problem. You could include the test scripts and doc files directly in the `spams` package folder by specifying them in the `MANIFEST.in` and using the `include_package_data` keyword ([https://setuptools.pypa.io/en/latest/userguide/datafiles.html](https://setuptools.pypa.io/en/latest/userguide/datafiles.html)), but first you need to move the `tests` and `doc` folder into the `spams` folder.
Might be caused by the setup.py trying to install the doc, I remember I axes that part. Since the manifest already puts them in the zip, I guess that part could be deleted, since absolutely no one will go look for the doc inside system folders (and it's in the zip, it just will not get installed).
I'll look into the doc setup (and remove the copy at install which is indeed useless).
The doc issue should be fixed by #14 

@daducci could you tell me if `pip install git+https://github.com/getspams/spams-python.git@remove_doc_copy` works for you?
No, it doesn't, I still get the error:
```
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/usr/local/tests'
Consider using the `--user` option or check the permissions.
```
**Note:** not fixed by #14 

@daducci could you give more details on the environment (OS, Python version, etc.) on which you are encountering this issue ?

And if you try installing previous version from PyPI like `pip install spams==2.6.3.3` ([here](https://pypi.org/project/spams/#history) is the history of version available on PyPI), all fail or some can be installed ?

Thanks
I'm on the new Macbook M1 with OSX 12.2.1 (Monterey) and python 3.9.10 (installed via `howebrew`), and this error happens with all `spams` versions. Whats is weird is that the first time I run `pip install spams` I get this error (and, in turn, our `dmri-amico` package is not installed), whereas if I run `pip install spams` a second time, the installation runs fine and also `dmri-amico` is properly installed without any error. Weird and I don't know why this happens...
Do you have an example of a package that you can install without any issue (beside `spams` or `dmri-amico`)?
I install all other packages, e.g. `numpy`, `scipy`, `nibabel`, `networkx` etc, via `pip install <name>` (i.e., without flag `--user`) and they are installed properly. May this be due to a different settings which installs the `tests` in a wrong folder?
Is `pip install spams-bin` working ?
Yes, indeed it does, without that error! :-)
Ok, I think I understand the problem. Package with pre-built wheels do not cause any problem, but when you need some local build from source, it causes the issue.
any modern local install should try to make wheels and install that, so I'd still point the culprit to the rest of the doc and test folder from the setup.py part. I'd guess the setup tries to install the files somehow (just like with the data folder) for important stuff, while the manifest only puts them in the source zip and discards everything from the real build.

Assuming nobody would ever read or run tests from the installed files and that we only want them in the zip (since people can easily refer to those instead), then this would be doable by removing all of those references from the setup.py and only using the manifest.in file. More details here https://packaging.python.org/en/latest/guides/using-manifest-in/

If you want to put here the wheel files before install (probably under /dist in your local folder), they likely contains references to those doc and tests files.
@samuelstjean I agree. I will add all the non-standard include to the `MANIFEST.in` and remove them from the `setup.py`.
@gdurif I'm pretty sure the problem is caused by the `data_files` keyword in `setup.py` trying to install the test scripts in a system folder (`/usr/local/`). `data_files` keyword is deprecated and does not work with wheels ([see here](https://setuptools.pypa.io/en/latest/references/keywords.html)), indeed `spams-bin` works because does not use it in it's `setup.py` file.

Remove the `data_files` section from `setup.py` should solve the problem. You could include the test scripts and doc files directly in the `spams` package folder by specifying them in the `MANIFEST.in` and using the `include_package_data` keyword ([https://setuptools.pypa.io/en/latest/userguide/datafiles.html](https://setuptools.pypa.io/en/latest/userguide/datafiles.html)), but first you need to move the `tests` and `doc` folder into the `spams` folder.

Fix patch:
diff --git a/MANIFEST.in b/MANIFEST.in
index 944ebba..400dae6 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -5,6 +5,10 @@ include version
 recursive-include spams_wrap *.h
 recursive-include spams_wrap *.cpp
 
+include spams/version
+
+recursive-include spams/tests *.py
+recursive-include spams/data *.png
 recursive-include tests *.py
 recursive-include data *.png
 recursive-include doc *
diff --git a/README.md b/README.md
index a04c1b8..3f49b09 100644
--- a/README.md
+++ b/README.md
@@ -49,9 +49,26 @@ Manipulated objects are imported from numpy and scipy. Matrices should be stored
 
 ### Testing the interface
 
+- From the command line (to be called from the project root directory):
 ```bash
-python tests/test_spams.py -h # to get help
-python tests/test_spams.py    # will run all the tests
+python tests/test_spams.py -h       # print the man page
+python tests/test_spams.py          # run all the tests
+```
+
+- From Python (assuming `spams` package is installed):
+```python
+from spams.tests import test_spams
+
+test_spams('-h')                    # print the man page
+test_spams()                        # run all tests
+test_spams(['sort', 'calcAAt'])     # run specific tests
+test_spams(python_exec='python3')   # specify the python exec
+```
+
+- From the command line (assuming `spams` package is installed):
+```bash
+# c.f. previous point for the different options
+python -c "from spams.tests import test_spams; test_spams()"
 ```
 
 ---
diff --git a/setup.py b/setup.py
index 8344486..561409a 100644
--- a/setup.py
+++ b/setup.py
@@ -2,7 +2,7 @@
 # import platform
 import sys
 
-from setuptools import setup, Extension, find_packages
+from setuptools import setup, Extension
 from setuptools.command.build_ext import build_ext
 
 from distutils.sysconfig import get_python_inc
@@ -193,18 +193,12 @@ def get_extension():
     python_requires='>=3',
     install_requires=['Cython>=0.29', 'numpy>=1.12',
                       'Pillow>=6.0', 'scipy>=1.0', 'six>=1.12'],
-    packages=find_packages(),
+    packages=['myscipy_rand', 'spams_wrap', 'spams', 'spams.tests'],
     cmdclass={'build_ext': CustomBuildExtCommand},
     ext_modules=get_extension(),
-    data_files=[('tests', ['tests/test_spams.py',
-                           'tests/test_decomp.py',
-                           'tests/test_dictLearn.py',
-                           'tests/test_linalg.py',
-                           'tests/test_prox.py',
-                           'tests/test_utils.py']),
-                ('doc', ['doc/doc_spams.pdf']),
-                ('tests', ['data/boat.png', 'data/lena.png'])],
-    include_package_data=True,
+    package_data={
+        "spams": ["data/*.png", "version"]
+    },
     zip_safe=True
 )
 
diff --git a/spams/data b/spams/data
new file mode 120000
index 0000000..4909e06
--- /dev/null
+++ b/spams/data
@@ -0,0 +1,1 @@
+../data
\ No newline at end of file
diff --git a/spams/version b/spams/version
new file mode 120000
index 0000000..1748d5b
--- /dev/null
+++ b/spams/version
@@ -0,0 +1,1 @@
+../version
\ No newline at end of file
diff --git a/version b/version
index 0ce6c9d..dacc32b 100644
--- a/version
+++ b/version
@@ -1,1 +1,1 @@
-2.6.5.1
+2.6.5.2


Test patch:
diff --git a/spams/tests b/spams/tests
new file mode 120000
index 0000000..6dd24e0
--- /dev/null
+++ b/spams/tests
@@ -0,0 +1,1 @@
+../tests
\ No newline at end of file
diff --git a/tests/__init__.py b/tests/__init__.py
new file mode 100644
index 0000000..23aa219
--- /dev/null
+++ b/tests/__init__.py
@@ -0,0 +1,1 @@
+from .run import test_spams
diff --git a/tests/run.py b/tests/run.py
new file mode 100644
index 0000000..1838ac7
--- /dev/null
+++ b/tests/run.py
@@ -0,0 +1,59 @@
+import subprocess
+import os
+
+def test_spams(args=None, python_exec = 'python'):
+    """Run spams-python tests from python
+    
+    To run the tests, from the command line, please check the README.md file
+    
+    Input:
+        args (string|string list): argument to pass to the the main test script,
+            use `args='-h'` for more details. If None (default), all tests are
+            run.
+        python_exec (string): python executable. By default, it is `'python'`.
+            Python 3+ is required. On some system, you may have to set 
+            `python_exec = 'python3'`.
+    """
+    
+    # check spams availability
+    try:
+        import spams
+    except:
+        raise ModuleNotFoundError("No module named 'spams'")
+    
+    # main test file
+    test_file = os.path.join("tests", "test_spams.py")
+    if os.path.isfile(test_file):
+        test_file = os.path.abspath(test_file)
+    else:
+        test_file = os.path.join(
+            os.path.dirname(os.path.abspath(spams.__file__)), test_file
+        )
+        
+    # test dir
+    test_dir = os.path.dirname(test_file)
+    
+    # check python executable
+    try:
+        python_version = subprocess.run([python_exec, "-V"], capture_output=True)
+    except:
+        raise SystemError(f"{python_exec} is not a valid python executable")
+    # check python version
+    if not "Python 3." in python_version.stdout.decode('UTF-8'):
+        raise SystemError("Python 3+ is required, try using python_exec = 'python3'")
+    
+    # check args
+    if args is None:
+        args = []
+    elif not (isinstance(args, str) or \
+        (isinstance(args, list) and \
+        all(isinstance(arg, str) for arg in args))):
+        raise TypeError("'args' input should be a string or a list of strings.")
+    
+    if isinstance(args, str):
+        args = [args]
+    
+    # run
+    subprocess.run([python_exec, 'test_spams.py'] + args, cwd = test_dir)
+    
+    
diff --git a/tests/test_dictLearn.py b/tests/test_dictLearn.py
index 88714a0..c6ea44a 100644
--- a/tests/test_dictLearn.py
+++ b/tests/test_dictLearn.py
@@ -1,6 +1,7 @@
 from __future__ import absolute_import, division, print_function
 
 import sys
+import os
 import numpy as np
 import scipy
 import scipy.sparse as ssp
@@ -15,6 +16,33 @@
     ssprand = myscipy_rand.rand
 else:
     ssprand = ssp.rand
+    
+    
+def get_img_file_path(img):
+    """Return path to an image file
+    
+    Arguments:
+        img (string): image filename without path among 'boat.png' or 
+        'lena.png'.
+        
+    Output:
+        img_file (string): normalized path to image input filename.
+    
+    """
+    # check input
+    if not img in ["boat.png", "lena.png"]:
+        raise ValueError("bad input, `img` should be 'boat.png' or 'lena.png'")
+    # try local file
+    img_file = os.path.join("data", img)
+    if os.path.isfile(img_file):
+        img_file = os.path.abspath(img_file)
+    else:
+        # file from install
+        img_file = os.path.join(
+            os.path.dirname(os.path.abspath(spams.__file__)), img_file
+        )
+    # output
+    return img_file
 
 
 def _extract_lasso_param(f_param):
@@ -46,8 +74,8 @@ def _objective(X, D, param, imgname=None):
 
 
 def test_trainDL():
-    img_file = 'data/boat.png'
     try:
+        img_file = get_img_file_path("boat.png")
         img = Image.open(img_file)
     except:
         print("Cannot load image %s : skipping test" % img_file)
@@ -140,8 +168,8 @@ def test_trainDL():
 
 
 def test_trainDL_Memory():
-    img_file = 'data/lena.png'
     try:
+        img_file = get_img_file_path("lena.png")
         img = Image.open(img_file)
     except:
         print("Cannot load image %s : skipping test" % img_file)
@@ -202,8 +230,8 @@ def test_trainDL_Memory():
 
 
 def test_structTrainDL():
-    img_file = 'data/lena.png'
     try:
+        img_file = get_img_file_path("lena.png")
         img = Image.open(img_file)
     except Exception as e:
         print("Cannot load image %s (%s) : skipping test" % (img_file, e))
@@ -376,8 +404,8 @@ def test_structTrainDL():
 
 
 def test_nmf():
-    img_file = 'data/boat.png'
     try:
+        img_file = get_img_file_path("boat.png")
         img = Image.open(img_file)
     except:
         print("Cannot load image %s : skipping test" % img_file)
@@ -413,8 +441,8 @@ def test_nmf():
 
 # Archetypal Analysis, run first steps with FISTA and run last steps with activeSet,
 def test_archetypalAnalysis():
-    img_file = 'data/lena.png'
     try:
+        img_file = get_img_file_path("lena.png")
         img = Image.open(img_file)
     except Exception as e:
         print("Cannot load image %s (%s) : skipping test" % (img_file, e))