Repository: iterative/dvc

Issue:
pip install errors: moto/boto3, awscli/colorama, requests?
> As noted in https://discordapp.com/channels/485586884165107732/565699007037571084/629449391266070563

I already have the repo cloned. I fetch and pull the latest master branch (to this moment) and freshly regenerate my virtualenv in `.env`, activate it and run `pip install -e ".[all,tests]" ` (as instructed in https://dvc.org/doc/user-guide/contributing). It succeeds but I get the following errors:

```console
Building wheels for collected packages: psutil
...
ERROR: moto 1.3.14.dev302 has requirement boto3>=1.9.201, but you'll have boto3 1.9.115 which is incompatible.
ERROR: awscli 1.16.252 has requirement colorama<=0.3.9,>=0.2.5, but you'll have colorama 0.4.1 which is incompatible.
```

pip install errors: moto/boto3, awscli/colorama, requests?
> As noted in https://discordapp.com/channels/485586884165107732/565699007037571084/629449391266070563

I already have the repo cloned. I fetch and pull the latest master branch (to this moment) and freshly regenerate my virtualenv in `.env`, activate it and run `pip install -e ".[all,tests]" ` (as instructed in https://dvc.org/doc/user-guide/contributing). It succeeds but I get the following errors:

```console
Building wheels for collected packages: psutil
...
ERROR: moto 1.3.14.dev302 has requirement boto3>=1.9.201, but you'll have boto3 1.9.115 which is incompatible.
ERROR: awscli 1.16.252 has requirement colorama<=0.3.9,>=0.2.5, but you'll have colorama 0.4.1 which is incompatible.
```



Discussion:
@jorgeorpinel 
What can you give some more details about your env? I am unable to reproduce on fresh `Python 3.7.4` environment,
tried:
`pip install -e ."[all,tests]"`
and
`pip install -e ."[all,tests]" --no-cache`

OK so I pull the latest version of master (`449f43f3` rn), I make sure no venv is active and remove my previous `.env` dir. Then:

```console
$ virtualenv --python python3 .env
...
$ source .env/bin/activate
...
$ pip install -e ".[all,tests]"  

Obtaining file:///.../dvc
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting requests>=2.22.0 (from dvc==0.61.2+449f43)
...
Collecting jsonpointer>=1.9 (from jsonpatch->cfn-lint>=0.4.0->moto>=1.3.14.dev55; extra == "tests"->dvc==0.61.2+449f43) ...
Building wheels for collected packages: pathspec, aws-sam-translator
  Building wheel for pathspec (setup.py) ... done
  ...
Successfully built pathspec aws-sam-translator
ERROR: awscli 1.16.252 has requirement colorama<=0.3.9,>=0.2.5, but you'll have colorama 0.4.1 which is incompatible.
ERROR: moto 1.3.14.dev312 has requirement boto3>=1.9.201, but you'll have boto3 1.9.115 which is incompatible.
talling collected packages: certifi, ...
$ which dvc
/.../dvc/.env/bin/dvc
$ dvc version             
DVC version: 0.60.1
Python version: 3.7.3
Platform: Darwin-18.7.0-x86_64-i386-64bit
Binary: False
Cache: reflink - True, hardlink - True, symlink - True
Filesystem type (cache directory): ('apfs', '/dev/disk1s1')
Filesystem type (workspace): ('apfs', '/dev/disk1s1')
```

> See full output for the pip install command in https://pastebin.com/f7c59V4n
BTW I just upgraded my system DVC with `pip3 install dvc --upgrade` and it worked but with 2 red lines:
```
docker-compose 1.23.2 has requirement requests!=2.11.0,!=2.12.2,!=2.18.0,<2.21,>=2.6.1, but you'll have requests 2.22.0 which is incompatible.
awsebcli 3.15.3 has requirement requests<2.21,>=2.20.1, but you'll have requests 2.22.0 which is incompatible.
```
which seems like a similar thing.
`dvc version` after update:
```console
DVC version: 0.61.2
Python version: 3.7.3
Platform: Darwin-18.7.0-x86_64-i386-64bit
Binary: False
Cache: reflink - True, hardlink - True, symlink - True
Filesystem type (cache directory): ('apfs', '/dev/disk1s1')
Filesystem type (workspace): ('apfs', '/dev/disk1s1')
```
> BTW I just upgraded my system DVC with pip3 install dvc --upgrade and it worked but with 2 red lines:

This one is about your previously installed packages, that require something else than what dvc requires. There is nothing we can do about it on our side.
And these

> ERROR: awscli 1.16.252 has requirement colorama<=0.3.9,>=0.2.5, but you'll have colorama 0.4.1 which is incompatible.
> ERROR: moto 1.3.14.dev312 has requirement boto3>=1.9.201, but you'll have boto3 1.9.115 which is incompatible.

are valid and are our fault, as our `all` reqs conflict with `test` reqs, which doesn't affect users, but does affect developers in a minor way. We should fix that, for sure.

> This one is about your previously installed packages, that require something else than what dvc requires. There is nothing we can do about it on our side.

How is it possible if it was installed in the clean `virtualenv`?
@shcheklein Apparently it wasn't clean, but rather inherited from the global env. `docker-compose` and `awsebcli` are definitely not our dependencies.
> How is it possible if it was installed in the clean virtualenv?

Ruslan is right about that one @shcheklein. In that https://github.com/iterative/dvc/issues/2570#issuecomment-538624781 I was referring to updating my system DVC installed as a package with `pip`, not to the development version installed as a git repo. Sorry for the confusion, we can safely ignore that comment and focus on https://github.com/iterative/dvc/issues/2570#issuecomment-538645844:

> are valid and are our fault, as our all reqs conflict with test reqs, which doesn't affect users, but does affect developers in a minor way. We should fix that, for sure.

p.s. do you mind if I hide https://github.com/iterative/dvc/issues/2570#issuecomment-538624781 and all its responses, off-topic and now resolved? 😋 
> p.s. do you mind if I hide #2570 (comment) and all its responses, off-topic and now resolved?

@jorgeorpinel Please don't hide comments. These are related to the topic.
is this still relevant?
@mroutis Yes, it is, I've seen it myself not that long ago.
@jorgeorpinel 
What can you give some more details about your env? I am unable to reproduce on fresh `Python 3.7.4` environment,
tried:
`pip install -e ."[all,tests]"`
and
`pip install -e ."[all,tests]" --no-cache`

OK so I pull the latest version of master (`449f43f3` rn), I make sure no venv is active and remove my previous `.env` dir. Then:

```console
$ virtualenv --python python3 .env
...
$ source .env/bin/activate
...
$ pip install -e ".[all,tests]"  

Obtaining file:///.../dvc
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting requests>=2.22.0 (from dvc==0.61.2+449f43)
...
Collecting jsonpointer>=1.9 (from jsonpatch->cfn-lint>=0.4.0->moto>=1.3.14.dev55; extra == "tests"->dvc==0.61.2+449f43) ...
Building wheels for collected packages: pathspec, aws-sam-translator
  Building wheel for pathspec (setup.py) ... done
  ...
Successfully built pathspec aws-sam-translator
ERROR: awscli 1.16.252 has requirement colorama<=0.3.9,>=0.2.5, but you'll have colorama 0.4.1 which is incompatible.
ERROR: moto 1.3.14.dev312 has requirement boto3>=1.9.201, but you'll have boto3 1.9.115 which is incompatible.
talling collected packages: certifi, ...
$ which dvc
/.../dvc/.env/bin/dvc
$ dvc version             
DVC version: 0.60.1
Python version: 3.7.3
Platform: Darwin-18.7.0-x86_64-i386-64bit
Binary: False
Cache: reflink - True, hardlink - True, symlink - True
Filesystem type (cache directory): ('apfs', '/dev/disk1s1')
Filesystem type (workspace): ('apfs', '/dev/disk1s1')
```

> See full output for the pip install command in https://pastebin.com/f7c59V4n
BTW I just upgraded my system DVC with `pip3 install dvc --upgrade` and it worked but with 2 red lines:
```
docker-compose 1.23.2 has requirement requests!=2.11.0,!=2.12.2,!=2.18.0,<2.21,>=2.6.1, but you'll have requests 2.22.0 which is incompatible.
awsebcli 3.15.3 has requirement requests<2.21,>=2.20.1, but you'll have requests 2.22.0 which is incompatible.
```
which seems like a similar thing.
`dvc version` after update:
```console
DVC version: 0.61.2
Python version: 3.7.3
Platform: Darwin-18.7.0-x86_64-i386-64bit
Binary: False
Cache: reflink - True, hardlink - True, symlink - True
Filesystem type (cache directory): ('apfs', '/dev/disk1s1')
Filesystem type (workspace): ('apfs', '/dev/disk1s1')
```
> BTW I just upgraded my system DVC with pip3 install dvc --upgrade and it worked but with 2 red lines:

This one is about your previously installed packages, that require something else than what dvc requires. There is nothing we can do about it on our side.
And these

> ERROR: awscli 1.16.252 has requirement colorama<=0.3.9,>=0.2.5, but you'll have colorama 0.4.1 which is incompatible.
> ERROR: moto 1.3.14.dev312 has requirement boto3>=1.9.201, but you'll have boto3 1.9.115 which is incompatible.

are valid and are our fault, as our `all` reqs conflict with `test` reqs, which doesn't affect users, but does affect developers in a minor way. We should fix that, for sure.

> This one is about your previously installed packages, that require something else than what dvc requires. There is nothing we can do about it on our side.

How is it possible if it was installed in the clean `virtualenv`?
@shcheklein Apparently it wasn't clean, but rather inherited from the global env. `docker-compose` and `awsebcli` are definitely not our dependencies.
> How is it possible if it was installed in the clean virtualenv?

Ruslan is right about that one @shcheklein. In that https://github.com/iterative/dvc/issues/2570#issuecomment-538624781 I was referring to updating my system DVC installed as a package with `pip`, not to the development version installed as a git repo. Sorry for the confusion, we can safely ignore that comment and focus on https://github.com/iterative/dvc/issues/2570#issuecomment-538645844:

> are valid and are our fault, as our all reqs conflict with test reqs, which doesn't affect users, but does affect developers in a minor way. We should fix that, for sure.

p.s. do you mind if I hide https://github.com/iterative/dvc/issues/2570#issuecomment-538624781 and all its responses, off-topic and now resolved? 😋 
> p.s. do you mind if I hide #2570 (comment) and all its responses, off-topic and now resolved?

@jorgeorpinel Please don't hide comments. These are related to the topic.
is this still relevant?
@mroutis Yes, it is, I've seen it myself not that long ago.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -52,6 +52,7 @@ def run(self):
 
 
 install_requires = [
+    "python-dateutil<2.8.1,>=2.1",  # Consolidates azure-blob-storage and boto3
     "ply>=3.9",  # See https://github.com/pyinstaller/pyinstaller/issues/1945
     "configparser>=3.5.0",
     "future>=0.16.0",
@@ -72,7 +73,7 @@ def run(self):
     "treelib>=1.5.5",
     "inflect>=2.1.0",
     "humanize>=0.5.1",
-    "PyYAML>=5.1.2",
+    "PyYAML>=5.1.2,<5.2",  # Compatibility with awscli
     "ruamel.yaml>=0.16.1",
     "funcy>=1.14",
     "pathspec>=0.6.0",
@@ -95,7 +96,7 @@ def run(self):
 # https://github.com/gsuitedevs/PyDrive/pull/180 into pydrive's master,
 # usage of google-api-python-client can be removed from DVC.
 gdrive = ["pydrive==1.3.1", "google-api-python-client>=1.2"]
-s3 = ["boto3==1.9.115"]
+s3 = ["boto3>=1.9.201"]
 azure = ["azure-storage-blob==2.1.0"]
 oss = ["oss2==2.6.1"]
 ssh = ["paramiko>=2.5.0"]
@@ -125,7 +126,7 @@ def run(self):
     "flaky>=3.5.3",
     "mock>=3.0.0",
     "xmltodict>=0.11.0",
-    "awscli==1.16.266",
+    "awscli>=1.16.297",
     "google-compute-engine==2.8.13",
     "Pygments",  # required by collective.checkdocs,
     "collective.checkdocs",


Test patch: