Repository: pypi/warehouse

Issue:
Run everything in containers
$ make reformat
# Create our Python 3.6 virtual environment
rm -rf .state/env
python3.6 -m venv .state/env
make: python3.6: No such file or directory
make: *** [.state/env/pyvenv.cfg] Error 1
Circular dependency in Makefile
From https://github.com/pypa/warehouse/pull/10278#issuecomment-974902013: 

> I believe this change has introduced a circular dependency, as perceived by these messages:
> 
> ```shellsession
> $ make -v | head -1
> GNU Make 3.81
> $ make build
> make[1]: Circular requirements/docs.txt <- .state/env/pyvenv.cfg dependency dropped.
> ...
> ```
> 
> For funsies, I tried the same with [remake](http://bashdb.sourceforge.net/remake/), which uses a newer version of `make`:
> 
> ```shellsession
> $ remake --version | head -1
> GNU Make 4.3+dbg-1.5
> $ remake build
> remake[1]: Entering directory '/Users/miketheman/workspace/miketheman/warehouse'
> remake[1]: Circular requirements/docs.txt <- .state/env/pyvenv.cfg dependency dropped.
> remake[1]: Circular requirements/lint.txt <- .state/env/pyvenv.cfg dependency dropped.
> remake[1]: '.state/docker-build' is up to date.
> ...
> ```
> 
> I don't know if this is harmful yet, but I'm having other issues getting started with development, and wondered if this is somehow related or not.

_Originally posted by @miketheman in https://github.com/pypa/warehouse/issues/10278#issuecomment-974902013_
Fix broken development setup
This removes circular dependency in Makefile added in #10278

Unblocks #10052
Fixes #10446
Fixes #10447


Discussion:
At the moment you'll need Python 3.6 installed on your machine to run this command. I'd recommend using [`pyenv`](https://github.com/pyenv/pyenv) to have multiple Python versions installed.

If you're interested, we might want to look into a way to move this into a Docker container instead, to avoid this issue in the future.
Yes, it would be great! Same for linting...
@di, I can work on this - should be pretty straightforward
I guess the question is whether you should be able to run stuff (especially build docs) without having to run docker. If that's the case, it makes sense to still preserve venv on the host machine. _Or_ it could be either separate target (IMO messy) or some kind of command line option or environment variable so you can select whether you want host or docker (or just default to docker if py3.6 is not found). Thoughts?
I think it's OK to assume that a Warehouse contributor has Docker installed. However, it's worth noting that Travis currently runs these commands (and the `test` command) without using a container -- [the way it does this is by invoking the `bin/tests` script directly instead of using the Makefile command](https://github.com/pypa/warehouse/blob/master/.travis.yml#L42), so it is necessary to "preserve" a way to run this without Docker, but it should only be necessary for CI.
I am also experiencing issues with this. I am unable to run `make lint` or `make reformat`

I manage my interpreters through `brew`. It recently moved me from 3.6  to 3.7. It would help me and other home-brew users as well if this was in the docker container. 
Now that #9083 is merged, the remaining things that aren't running in containers are:

- [ ] [`make docs`](https://github.com/pypa/warehouse/blob/961575f8405566f8d567d51a0ea928c7a9eecb63/Makefile#L114-L116)
- [ ] [`make deps`](https://github.com/pypa/warehouse/blob/961575f8405566f8d567d51a0ea928c7a9eecb63/Makefile#L122-L131)
- [ ] [`make translations`](https://github.com/pypa/warehouse/blob/961575f8405566f8d567d51a0ea928c7a9eecb63/Makefile#L175-L209) (and it's subcommands)
Reposting my comment:
I think we could mostly solve the problem by making the targets more granular, but at core we need to solve something: in order to install pip-compile, we need to know which version to install it too, which means we need to run pip-compile and so on.
If we install pip-compile without the hashes, it kinda defeats the purpose of using hashes in the first place.

So one solution could be to not disclose to `make` that the pip-compile target needs the pip-compile requirement file, considering that it doesn't often change, and the venv gets recreated once in a while anyway.

So we would have:
- A target for venv creation but not installation of dependencies
- a target for pip-tools that would depend on that one, which would actually use the requirements/pip-tools.txt file but wouldn't say it
- requirements/*.txt would depend on that one
- the full environement target, the one everyone uses, would use the relevant .txt files and declare it.

I believe it would work.
Or, we could take this as an opportunity to move more towards #4948, since it doesn't make a ton of sense right now for us to compile dependencies on one platform (the host OS) to be used on a different platform (inside the container images), which removes the need to build dependencies in the virtual environment.
Why not to vendorize `pip-compile`?
For example with `pex`.

```python
pip install pex
pex pip-tools --console-script pip-compile --output-file pip-compile.pex
```
```
-rwxr-xr-x 1 root root 4750609 Dec  7 10:53 pip-compile.pex
```
@di @ewdurbin please review.
@ewjoachim addressed all comments. PTAL.
@di @ewjoachim @ewdurbin ping?

Resolving patch:
diff --git a/bin/depchecker.py b/bin/depchecker.py
new file mode 100644
--- /dev/null
+++ b/bin/depchecker.py
@@ -0,0 +1,33 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.import sys
+
+import sys
+
+from pip_api import parse_requirements
+
+left, right = sys.argv[1:3]
+left_reqs = parse_requirements(left).keys()
+right_reqs = parse_requirements(right).keys()
+
+extra_in_left = left_reqs - right_reqs
+extra_in_right = right_reqs - left_reqs
+
+if extra_in_left:
+    for dep in sorted(extra_in_left):
+        print("- {}".format(dep))
+
+if extra_in_right:
+    for dep in sorted(extra_in_right):
+        print("+ {}".format(dep))
+
+if extra_in_left or extra_in_right:
+    sys.exit(1)


Test patch:
diff --git a/bin/tests b/bin/tests
--- a/bin/tests
+++ b/bin/tests
@@ -7,7 +7,7 @@ set -e
 export LC_ALL="${ENCODING:-en_US.UTF-8}"
 export LANG="${ENCODING:-en_US.UTF-8}"
 
-COMMAND_ARGS=$@
+COMMAND_ARGS="$@"
 
 # Test the postgres connection
 while [ $# -gt 0 ]; do