Repository: xonsh/xonsh

Issue:
github workflow refactoring
<!--- Provide a general summary of the issue in the Title above -->
<!--- If you have a question along the lines of "How do I do this Bash command in xonsh"
please first look over the Bash to Xonsh translation guide: https://xon.sh/bash_to_xsh.html
If you don't find an answer there, please do open an issue! -->

## xonfig

<details>

```
+------------------+----------------------+
| xonsh            | 0.9.22               |
| Git SHA          | 00a42e01             |
| Commit Date      | Sep 18 19:57:59 2020 |
| Python           | 3.8.5                |
| PLY              | 3.11                 |
| have readline    | True                 |
| prompt toolkit   | 3.0.3                |
| shell type       | prompt_toolkit       |
| pygments         | 2.7.1                |
| on posix         | True                 |
| on linux         | True                 |
| distro           | manjaro              |
| on darwin        | False                |
| on windows       | False                |
| on cygwin        | False                |
| on msys2         | False                |
| is superuser     | False                |
| default encoding | utf-8                |
| xonsh encoding   | utf-8                |
| encoding errors  | surrogateescape      |
| on jupyter       | False                |
| jupyter kernel   | None                 |
+------------------+----------------------+

```

</details>


## Current Behavior
<!--- Tell us what happens instead of the expected behavior -->
<!--- If part of your bug report is a traceback, please first enter debug mode before triggering the error
To enter debug mode, set the environment variable `XONSH_DEBUG=1` _before_ starting `xonsh`.
On Linux and OSX, an easy way to to do this is to run `env XONSH_DEBUG=1 xonsh` -->

I am curious about knowing the current setup of the github workflow. Any of the core members please clarify. I am ready to start working on them 

1. having multiple workflow files
  - there are 3 platforms * 3 py versions 
  - issue: upon failure it sends multiple emails. it is annoying. 
  - single file with matrix would suffice. I want to know why there are multiple files and why it needs generation.
2. conda on ci and requirements file splitup
  - the dependencies may diverge and we will never know. the local testing will not be the same as ci
  - can't we use tox with requirements/poetry/pipenv 
  - single set of dependency declaration will surely help. I suggest we use poetry+tox+tox-gh-actions to run the same setup on local and ci server
3. black versions 
  - in the precommit it uses pre-built hook which uses old version
  - in the requirements file, no version is mentioned
  - in the ci it uses pre-built action
  - suggesion: can't we just use a single version all across the systems. it is a python package. it pre-commit we can configure to run local package. the same can be used in ci as well. so a single version across systems and no confusions.


## Expected Behavior
<!--- Tell us what should happen -->

## For community
⬇️  **Please click the 👍 reaction instead of leaving a `+1` or 👍  comment**



Discussion:
@scopatz 
Hi @jnoortheen -- improvements to the CI are always welcome.  I'll try to answer a few of your questions

> * issue: upon failure it sends multiple emails. it is annoying.

You can turn off those emails.

> * single file with matrix would suffice. I want to know why there are multiple files and why it needs generation.

That's how I set it up initially, but github actions doesn't (or didn't) let you restart individual runs if one of them failed, so we were constantly re-running all jobs when just one of them failed, so we removed the OS dimension from the matrix.  If github fixes this, then a matrix run would be great.

> * the dependencies may diverge and we will never know. the local testing will not be the same as ci

So @scopatz and I and probably others use `conda` everywhere, so our local testing very much matches our CI.

`tox` is great, but all of this assumes that everyone uses `pip` for everything, and that's just not the case.  so setting things up to use `tox` is ok, but we also want to make sure `conda` dependencies are working.  Also, I do _not_ like `pipenv` and don't want it anywhere near our CI setup. `poetry` is pretty solid.

> * in the precommit it uses pre-built hook which uses old version

That should be updated

> * in the requirements file, no version is mentioned

I think we should maybe remove the requirements file entirely in favor of various `extras_` in the `setup.py` file for the `pip` crowd.

> * suggesion: can't we just use a single version all across the systems. it is a python package. it pre-commit we can configure to run local package. the same can be used in ci as well. so a single version across systems and no confusions.

I think that's a fine idea.
Thanks @gforsyth for the clarification. 

2. so can I start setup poetry ? with dephell we can manage the vendoring as well. What do you think ? we can also keep using conda but will manage dependencies with poetry. 

3. Ok I will rewrite pre-commit with local version. 
@jnoortheen the changes to pre-commit sound good to me

in re: poetry, I have nothing against it, I'm just unclear what it's bringing to the table here.  We have very limited dependencies.
Is there a big benefit to switching over to another package framework?
1. managing development dependencies and locking their versions 
2. managing the virtual environment. 
3. single place to specify the deps. now it is reqs/environment file(3 of them). 
the tools like dependabot can integrated as well. 
Thanks for laying that out, @jnoortheen!  I have a few other thoughts

> managing development dependencies and locking their versions

Do we want to lock these versions?  We generally want to support as wide a variety of dependencies as possible to make `xonsh` widely usable.  I know `black` has breaking changes periodically, but are there other dev dependencies that we need this for?

> single place to specify the deps. now it is reqs/environment file(3 of them).

Either way, we can probably remove `tests-35.txt`.  Fairly certain that isn't being used.
I dont think the wide support is a valid point. we can skip adding the lock file to the repo if you feel strong about it. it is not only about black, the flake8 plugins, pytest versions may break tests, mypy versions will all breaking too. we are not excluding any sort of Python version users. all the versions supported by the xonsh can be specified in that file too. 

   > Either way, we can probably remove tests-35.txt. Fairly certain that isn't being used.


even then managing reqs is a pain, adding new dependencies and not forgetting to add to the requirements file. Please give it a try yourself. 

Resolving patch:
diff --git a/xonsh/tools.py b/xonsh/tools.py
--- a/xonsh/tools.py
+++ b/xonsh/tools.py
@@ -727,7 +727,6 @@ def dec(obj):
 # See the Python software license: https://docs.python.org/3/license.html
 # Copyright (c) Python Software Foundation. All rights reserved.
 class _RedirectStream:
-
     _stream: tp.Optional[str] = None
 
     def __init__(self, new_target):
@@ -1296,7 +1295,7 @@ def to_logfile_opt(x):
     the filepath if it is a writable file or None if the filepath is not
     valid, informing the user on stderr about the invalid choice.
     """
-    if isinstance(x, os.PathLike):
+    if isinstance(x, os.PathLike):  # type: ignore
         x = str(x)
     if is_logfile_opt(x):
         return x


Test patch:
diff --git a/requirements/tests.txt b/requirements/tests.txt
--- a/requirements/tests.txt
+++ b/requirements/tests.txt
@@ -10,4 +10,4 @@ codecov
 coverage
 black==19.10b0 --pre
 pre-commit
-mypy>=0.782
+mypy==0.790