Repository: privacyidea/privacyidea

Issue:
Deterministic installations with pinned dependencies
Currently, this is just an idea and I would be interested in your opinions :-)

There are currently three ways to install privacyIDEA (please correct me if I'm wrong):
* via native distribution packages, currently only Ubuntu
* via distribution packages using virtualenv (e.g. centos and debian-virtualenv)
* manually in a virtualenv via ``pip install privacyidea``

In the past, we often had problems when there were new versions of dependencies which were incompatible with our codebase (e.g. ldap3/pyasn #912).

This can still happen, e.g. assume there is a dependency ``X`` which releases a new version ``1.1`` today that is incompatible with privacyIDEA 2.22. Then, all users who use ``pip install privacyidea`` today to install privacyIDEA 2.22 will get the newest version ``1.1`` of dependency ``X``, which will cause problems, and will need to manually downgrade ``X`` to get a working installation. We can react by putting ``X<1.1`` in our ``setup.py``, but this requires us to make a new release.

In case of Ubuntu packages, the package repositories don't get updated too often, so the risk of incompatible new versions is pretty low here. The risk is higher for virtualenv packages and pip installations, because ``privacyidea-pip-update`` will upgrade all installed dependencies to their latest versions by default (though we can still explicitly exclude versions in ``setup.py``).

So it might be nice to have "deterministic" installations in the sense that all versions of dependencies installed on user's machines are pinned and known to work with our current stable version.

Currently, I'm not sure how to do that nicely: Putting pinned dependencies in ``setup.py`` is [discouraged](https://packaging.python.org/discussions/install-requires-vs-requirements/#install-requires). ``requirements.txt`` contains pinned versions, but ``pip install privacyidea`` installs dependencies from setup.py, not requirement.txt.

What do you think? :)


Discussion:
I think actually, this problem mostly affects pip installations:
* For our Ubuntu packages, the risk of newer dependency versions breaking privacyIDEA is low because new versions are added to the package repositories pretty slowly.
* For our virtualenv-based (CentOS) packages, we ship a complete virtualenv. We can control the shipped dependency versions when building the packages. These installations _shouldn't be updated via privacyidea-pip-update_, but only via the distribution package manager (is that correct?).
* pip installations _do_ have this problem, because ``privacyidea-pip-update`` updates all installed dependencies. How about we supply a ``stable-requirements.txt`` somewhere (e.g. on github?) which is updated with every stable release and pins the versions of all dependencies. ``privacyidea-pip-update`` could then just fetch this file and pass it to ``pip install -r``. This way, we wouldn't blindly update *all* installed dependencies to their latest versions.
- problem with packages is that it is not isolated environment, so you very easily can end up with conflicting with other app, we had several times problems conflicting python apps e.g. with ansible...

for that best practises:
It is not considered best practice to use install_requires to pin dependencies to specific versions, or to specify sub-dependencies (i.e. dependencies of your dependencies). This is overly-restrictive, and prevents the user from gaining the benefit of dependency upgrades.
- i strongly disagree with this, in practise this causes a LOT OF problems (we had also problems with cerbot) and i would say that benefit of upgrades is lower than breaking functionality without any notice, of course it is then responsibility of developers to upgrade, but at least they know that they upgraded dependencies and bugs can be catched before release...
Thank you for your thoughts!
> problem with packages is that it is not isolated environment, so you very easily can end up with conflicting with other app, we had several times problems conflicting python apps e.g. with ansible...

Do you mean distribution packages (like Ubuntu packages) here? i.e. conflicts because other apps installed on the same machine need different versions of the same dependencies as privacyIDEA?
If think the easiest way to get a reliable approach is to provide a "requirements-stable.txt".

This way the administrator could choose if he wants to get the latest and greatest packages and run a 

     privacyidea-pip-update

or if he wants the stable (older packages) and run

     privacyidea-pip-update --stable

or
     privacyidea-pip-update --stable --stable-file my-own-stable-file-since-I-know-it-better.txt

with the new parameter "--stable" ;-), which would run 

     pip install -r requirement-stable.txt

or

     pip install -r my-own-stable-file-since-I-know-it-better.txt

After all it is the sense of a virtual env to be able to provide a specific combination of versions.
This way we can gain everything and loose nothing.

The "privacyidea-pip-update --stable" of course would be a bit more complicated: check for a new privacyidea version, get it, get the requirments stable from this and pin to those versions.
I like the idea of a ``--stable`` parameter to ``privacyidea-pip-update``!
But we would still have a problem for new installations. If someone enters ``pip install privacyidea``, the package is downloaded and the requirements are read from the ``setup.py`` which uses ``>=``, which will result in the newest versions of dependencies installed. And they will be downgraded only when ``privacyidea-pip-update --stable`` is used :-/

Just an idea to fix this, I'm not sure if it's a very good idea:
* In our master branch, we provide a ``stable-requirements.txt`` with pinned dependencies. This includes the current version of privacyIDEA, (e.g. 2.23) itself (``privacyidea==2.23``).
* We change the installation instructions of pip installations to ``pip install -r https://raw.githubusercontent.com/privacyidea/privacyidea/master/stable-requirements.txt``. This will install the pinned dependencies and privacyIDEA 2.23 itself.
* Once 2.24 is out, we update the ``stable-requirements.txt`` (changing the privacyIDEA line to ``privacyidea==2.24``, and updating the dependency versions), and ``privacyidea-pip-update`` just runs the same ``pip install -r`` command as above! privacyIDEA itself and all dependencies are updated.

Good things about this:
* pip installations would not break on updates due to updated dependencies anymore
* People who want the newest versions of everything could still use ``pip install privacyidea``.
* We could use the same ``stable-requirements.txt`` to build virtualenv-based distribution packages

Bad things about this:
* We rely on pypi *and* github being available all the time
* We assume that the ``stable-requirements.txt`` is never compromised (reminds me of [this story](https://medium.com/@vesirin/how-i-gained-commit-access-to-homebrew-in-30-minutes-2ae314df03ab))
If s.o. installs via pip, we always rely on pypi.
To avoid relying on github, we could mirror the ``stable-requirements.txt`` to ` ``https://privacyidea.org/stable-requirements.txt`` during our release process.
This would also shorten the necessary typing.

I like it.
Just an idea to support the "latest and greatest"-users (and to become aware of possible conflicts):
We could set up a nightly installation run of the privacyIDAEA master, one with `setup.py` and one with `pip install requirements.txt` and run some integration tests on them to check for possible conflicts.
@fredreichbier After giving it a rethink I am not sure, if we really should pin all versions! This would mean it becomes our responsibility to check if the pinned version has any security issues.

There might be a low hanging fruit based on the idea by @plettich .
We have our .travis.yml, that defines the test runs: https://github.com/privacyidea/privacyidea/blob/master/.travis.yml
This is run on every commit to the github repo.

We could run another test run, that actually tests with the latest updates of a complete virtualenv.
Maybe we could use the travis-ci cronjobs? https://docs.travis-ci.com/user/cron-jobs/

We would need to check, if we can have a matrix in the ``install`` section, which would result in two test runs, one with the pinned ``requirements.txt`` and one with the updated dependencies.


A periodic CI job that pulls the latest versions of all dependencies and runs our tests sounds like a good idea to me!

Still, I think in the long run it would be a good idea to better "equalize" the various installation methods. Also because for the virtualenv-based distribution packages, we already have the responsibility to check if the installed versions have any security issues :-/
Just found [this](https://pypi.org/project/pbr/) little python package which pulls the `requirements.txt` into `setup.py`: https://docs.openstack.org/pbr/latest/user/features.html#requirements
Nice, who wants to do either
* check possibilities at travis (I think they are limited)
or 
* investigate what tools to use?
* setup a system/machine that does this job?
just note, what about using pipenv, it is officially recommended tool https://packaging.python.org/tutorials/managing-dependencies/#managing-dependencies and as i checked it should manage also dependencies of dependencies, it has .lock file...
@fredreichbier @plettich We cleared our requirements.txt. We can use this to run ``pip install -r requirements``.

Starting with version 3.0 we will have xenial and bionic packages with this very virtualenv. We will have CentOS packages with this env.
So I assume we can close this?
I think we're very close to closing this :) I have one remaining question: What do we do with ``privacyidea-pip-update``? Should we keep it around, even though it wouldn't be the preferred method to upgrade a virtualenv?
So what would then be the preferred method to upgrade, if not privacyidea-pip-update?
This script still does:
1. upgrade privacyidea
2. upgrade the dependencies to the correct version
3. run schema upgrade of DB

I think this is an important script, if privacyidea was *not* installed via deb or rpm.
I think in the current version, privacyidea-pip-update upgrades *all* packages in the virtualenv to their latest version. This would mean that we end up with versions that are newer than the versions in our ``requirements.txt`` and which are "unsupported".
I think the script would rather have to fetch the new ``requirements.txt`` somehow and do ``pip install -r requirements.txt``?
Oh, you are right! I also need the requirements.txt for the initial install!
We can actually put the requirements.txt file as data in setup.py, so that it gets installed and we can use it to run a 

   pip install -r /opt/privacyidea/lib/requirements/requirements.txt

afterwards. 
Hm I guess this could work! Could happen that we install version X of some package when doing ``python setup.py install`` and install some other version Y of the same package when doing ``pip install -r requirements.txt`` afterwards, but I guess this isn't that critical.

But I was wondering: What about just doing
```
pip install -r https://raw.githubusercontent.com/privacyidea/privacyidea/v3.0dev7/requirements.txt
```
for upgrading to v3.0dev7?
But that's possibly bad style because it is kind of like the ``curl ... | bash`` pattern?

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -145,7 +145,8 @@ def get_scripts(dir):
                   "migrations/README",
                   "migrations/script.py.mako"]),
                 ('lib/privacyidea/migrations/versions',
-                 get_file_list("migrations/versions/"))
+                 get_file_list("migrations/versions/")),
+                ('lib/privacyidea/', ['requirements.txt'])
                 ],
     classifiers=["Framework :: Flask",
                  "License :: OSI Approved :: "


Test patch: