Repository: pandas-dev/pandas
Pull request: CI: Fixes to conda environment names
- [x] closes #23851 
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Problem statement:
CLN: Fixes to conda environment names
We have some inconsistency regarding the names of the conda environment. At the moment, all environments for the CI are named `pandas`, and the environment created locally following the set up instructions (and using `environment.yml`) is named `pandas-dev`.

Then, inside several scripts (e.g. `ci/code_checks.sh`) we do a `source activate pandas`. Which means that the script works well in CI jobs, but it fails locally unless we also have a `pandas` environment locally.

Another thing is that in the CI config we use a `CONDA_ENV` environment variable, and we ignore the conda environment names in the yaml files.

What I would do is:
- Use `pandas-dev` as the environment name in all our conda yaml files
- Create the environments with the environment name in the used yaml file (do not override with `-n $CONDA_ENV` as we do now)
- Do not use `source activate` inside the scripts (use it in the steps in the CI config files when needed). This way when running scripts locally, the current environment will be used

@jreback @TomAugspurger @jorisvandenbossche any objection?

Hints from issue discussion:
this is ok
i will take a look!
Thanks @charlesdong1991, let me know if you need clarifications or help.
Hi, @datapythonista , sorry, i think i do need your help, following your explanation above, i did a small modification to change pandas to pandas-dev, not sure if it's the right place to do... and appreciate a lot if you could give more clarifications
The main changes required are:
- In `ci/deps/*.yaml` change the name of the environments from `pandas` to `pandas-dev`
- Do a grep for `CONDA_ENV`, and remove the `-n $CONDA_ENV` in the `conda env install` calls. And remove the definitions of it, as it won't be used anymore.
- Do a grep for `source activate` and move the activations inside the scripts (mainly in `ci/`) to the CI configuration files, in the place before those scripts are called

For example, in `ci/incremental/build.sh` there is a `source activate $CONDA_ENV`. This should be removed from there, so if we call the script in our machines the active environment is used. And for the CI to work, a `source activate pandas-dev` should be placed in `.travis.yml` before the call to `build.sh`, so the environment is active when that is called.
I'd like to add one more location to this: `doc/source/style.ipynb` 
For compiling the docs, I always have to change the name of the kernel in there, to make ipython use the right one:
```diff
  "metadata": {
   "kernelspec": {
    "display_name": "Python 3",
    "language": "python",
+   "name": "pandas-dev"
-   "name": "python3"
  },
```
You should be able to fix that with a jupyter kernelspec command. Kernel names don’t have to be tied to environment names, and I think python3 is the default.


________________________________
From: h-vetinari <notifications@github.com>
Sent: Thursday, November 22, 2018 10:28
To: pandas-dev/pandas
Cc: Tom Augspurger; Mention
Subject: Re: [pandas-dev/pandas] CLN: Fixes to conda environment names (#23851)


I'd like to add one more location to this: doc/source/style.ipynb
For compiling the docs, I always have to change the name of the kernel in there, to make ipython use the right one:

  "metadata": {
   "kernelspec": {
    "display_name": "Python 3",
    "language": "python",
+   "name": "pandas-dev"
-   "name": "python3"
  },

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/pandas-dev/pandas/issues/23851#issuecomment-441078338>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABQHIqoDu8ezsDhx5buZmCRhMElxq13qks5uxtCegaJpZM4Yueaf>.

@TomAugspurger Yes, of course, but I don't want my `python3` kernel spec to point to the dev environment. If we're already talking environment names (and the default from `environment.yml`), I think this is a reasonable adaptation.

Commit message: debugging

Patch to review:
diff --git a/.travis.yml b/.travis.yml
index 6d31adcbf8a43..a1f69e8502a85 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -105,6 +105,7 @@ before_script:
 
 script:
   - echo "script start"
+  - source activate pandas-dev
   - ci/run_build_docs.sh
   - ci/script_single.sh
   - ci/script_multi.sh
diff --git a/ci/azure/linux.yml b/ci/azure/linux.yml
index b5a8e36d5097d..cd5d4c07598c7 100644
--- a/ci/azure/linux.yml
+++ b/ci/azure/linux.yml
@@ -35,21 +35,24 @@ jobs:
         ci/incremental/install_miniconda.sh
         export PATH=$HOME/miniconda3/bin:$PATH
         echo "Setting up Conda environment"
+        source activate pandas-dev
         ci/incremental/setup_conda_environment.sh
       displayName: 'Before Install'
     - script: |
         export PATH=$HOME/miniconda3/bin:$PATH
+        source activate pandas-dev
         ci/incremental/build.sh
       displayName: 'Build'
     - script: |
         export PATH=$HOME/miniconda3/bin:$PATH
+        source activate pandas-dev
         ci/script_single.sh
         ci/script_multi.sh
         echo "[Test done]"
       displayName: 'Test'
     - script: |
         export PATH=$HOME/miniconda3/bin:$PATH
-        source activate pandas && pushd /tmp && python -c "import pandas; pandas.show_versions();" && popd
+        source activate pandas-dev && pushd /tmp && python -c "import pandas; pandas.show_versions();" && popd
     - task: PublishTestResults@2
       inputs:
         testResultsFiles: 'test-data-*.xml'
diff --git a/ci/azure/macos.yml b/ci/azure/macos.yml
index 16f2fa2d4890f..2cb39ba063d25 100644
--- a/ci/azure/macos.yml
+++ b/ci/azure/macos.yml
@@ -22,21 +22,24 @@ jobs:
         ci/incremental/install_miniconda.sh
         export PATH=$HOME/miniconda3/bin:$PATH
         echo "Setting up Conda environment"
+        source activate pandas-dev
         ci/incremental/setup_conda_environment.sh
       displayName: 'Before Install'
     - script: |
         export PATH=$HOME/miniconda3/bin:$PATH
+        source activate pandas-dev
         ci/incremental/build.sh
       displayName: 'Build'
     - script: |
         export PATH=$HOME/miniconda3/bin:$PATH
+        source activate pandas-dev
         ci/script_single.sh
         ci/script_multi.sh
         echo "[Test done]"
       displayName: 'Test'
     - script: |
         export PATH=$HOME/miniconda3/bin:$PATH
-        source activate pandas && pushd /tmp && python -c "import pandas; pandas.show_versions();" && popd
+        source activate pandas-dev && pushd /tmp && python -c "import pandas; pandas.show_versions();" && popd
     - task: PublishTestResults@2
       inputs:
         testResultsFiles: 'test-data-*.xml'
diff --git a/ci/build_docs.sh b/ci/build_docs.sh
index 33340a1c038dc..f89c4369dff4a 100755
--- a/ci/build_docs.sh
+++ b/ci/build_docs.sh
@@ -12,8 +12,6 @@ if [ "$DOC" ]; then
 
     echo "Will build docs"
 
-    source activate pandas
-
     echo ###############################
     echo # Log file for the doc build  #
     echo ###############################
diff --git a/ci/deps/azure-27-compat.yaml b/ci/deps/azure-27-compat.yaml
index 44c561e9c8911..f3cc615c35243 100644
--- a/ci/deps/azure-27-compat.yaml
+++ b/ci/deps/azure-27-compat.yaml
@@ -1,4 +1,4 @@
-name: pandas
+name: pandas-dev
 channels:
   - defaults
   - conda-forge
diff --git a/ci/deps/azure-36-locale_slow.yaml b/ci/deps/azure-36-locale_slow.yaml
index 7e40bd1a9979e..4bbc6a2c11f1e 100644
--- a/ci/deps/azure-36-locale_slow.yaml
+++ b/ci/deps/azure-36-locale_slow.yaml
@@ -1,4 +1,4 @@
-name: pandas
+name: pandas-dev
 channels:
   - defaults
   - conda-forge
diff --git a/ci/deps/azure-37-locale.yaml b/ci/deps/azure-37-locale.yaml
index 59c8818eaef1e..2b38465c04512 100644
--- a/ci/deps/azure-37-locale.yaml
+++ b/ci/deps/azure-37-locale.yaml
@@ -1,4 +1,4 @@
-name: pandas
+name: pandas-dev
 channels:
   - defaults
   - conda-forge
diff --git a/ci/deps/azure-macos-35.yaml b/ci/deps/azure-macos-35.yaml
index 6ccdc79d11b27..7a0c3b81ac8f9 100644
--- a/ci/deps/azure-macos-35.yaml
+++ b/ci/deps/azure-macos-35.yaml
@@ -1,4 +1,4 @@
-name: pandas
+name: pandas-dev
 channels:
   - defaults
 dependencies:
diff --git a/ci/deps/azure-windows-27.yaml b/ci/deps/azure-windows-27.yaml
index dc68129a5e6d3..b1533b071fa74 100644
--- a/ci/deps/azure-windows-27.yaml
+++ b/ci/deps/azure-windows-27.yaml
@@ -1,4 +1,4 @@
-name: pandas
+name: pandas-dev
 channels:
   - defaults
   - conda-forge
diff --git a/ci/deps/azure-windows-36.yaml b/ci/deps/azure-windows-36.yaml
index af42545af7971..817aab66c65aa 100644
--- a/ci/deps/azure-windows-36.yaml
+++ b/ci/deps/azure-windows-36.yaml
@@ -1,4 +1,4 @@
-name: pandas
+name: pandas-dev
 channels:
   - defaults
   - conda-forge
diff --git a/ci/deps/circle-36-locale.yaml b/ci/deps/circle-36-locale.yaml
index 59c8818eaef1e..2b38465c04512 100644
--- a/ci/deps/circle-36-locale.yaml
+++ b/ci/deps/circle-36-locale.yaml
@@ -1,4 +1,4 @@
-name: pandas
+name: pandas-dev
 channels:
   - defaults
   - conda-forge
diff --git a/ci/deps/travis-27-locale.yaml b/ci/deps/travis-27-locale.yaml
index c8d17cf190e35..0846ef5e8264e 100644
--- a/ci/deps/travis-27-locale.yaml
+++ b/ci/deps/travis-27-locale.yaml
@@ -1,4 +1,4 @@
-name: pandas
+name: pandas-dev
 channels:
   - defaults
   - conda-forge
diff --git a/ci/deps/travis-27.yaml b/ci/deps/travis-27.yaml
index 5a9e206ec2c69..8d14673ebde6d 100644
--- a/ci/deps/travis-27.yaml
+++ b/ci/deps/travis-27.yaml
@@ -1,4 +1,4 @@
-name: pandas
+name: pandas-dev
 channels:
   - defaults
   - conda-forge
diff --git a/ci/deps/travis-36-doc.yaml b/ci/deps/travis-36-doc.yaml
index fb54c784d6fac..ed0764fab414a 100644
--- a/ci/deps/travis-36-doc.yaml
+++ b/ci/deps/travis-36-doc.yaml
@@ -1,4 +1,4 @@
-name: pandas
+name: pandas-dev
 channels:
   - defaults
   - conda-forge
diff --git a/ci/deps/travis-36-slow.yaml b/ci/deps/travis-36-slow.yaml
index 3157ecac3a902..a6ffdb95e5e7c 100644
--- a/ci/deps/travis-36-slow.yaml
+++ b/ci/deps/travis-36-slow.yaml
@@ -1,4 +1,4 @@
-name: pandas
+name: pandas-dev
 channels:
   - defaults
   - conda-forge
diff --git a/ci/deps/travis-36.yaml b/ci/deps/travis-36.yaml
index 1880fa2501581..1781f67041f44 100644
--- a/ci/deps/travis-36.yaml
+++ b/ci/deps/travis-36.yaml
@@ -1,4 +1,4 @@
-name: pandas
+name: pandas-dev
 channels:
   - defaults
   - conda-forge
diff --git a/ci/deps/travis-37-numpydev.yaml b/ci/deps/travis-37-numpydev.yaml
index 82c75b7c91b1f..99ae228f25de3 100644
--- a/ci/deps/travis-37-numpydev.yaml
+++ b/ci/deps/travis-37-numpydev.yaml
@@ -1,4 +1,4 @@
-name: pandas
+name: pandas-dev
 channels:
   - defaults
 dependencies:
diff --git a/ci/deps/travis-37.yaml b/ci/deps/travis-37.yaml
index 7dbd85ac27df6..a297786f6b14d 100644
--- a/ci/deps/travis-37.yaml
+++ b/ci/deps/travis-37.yaml
@@ -1,4 +1,4 @@
-name: pandas
+name: pandas-dev
 channels:
   - defaults
   - conda-forge
diff --git a/ci/incremental/build.cmd b/ci/incremental/build.cmd
index d2fd06d7d9e50..7115dea8c7eac 100644
--- a/ci/incremental/build.cmd
+++ b/ci/incremental/build.cmd
@@ -1,5 +1,5 @@
 @rem https://github.com/numba/numba/blob/master/buildscripts/incremental/build.cmd
-call activate %CONDA_ENV%
+
 
 @rem Build numba extensions without silencing compile errors
 python setup.py build_ext -q --inplace
diff --git a/ci/incremental/build.sh b/ci/incremental/build.sh
index 8f2301a3b7ef5..05648037935a3 100755
--- a/ci/incremental/build.sh
+++ b/ci/incremental/build.sh
@@ -1,7 +1,5 @@
 #!/bin/bash
 
-source activate $CONDA_ENV
-
 # Make sure any error below is reported as such
 set -v -e
 
diff --git a/ci/incremental/setup_conda_environment.cmd b/ci/incremental/setup_conda_environment.cmd
index 35595ffb03695..12353ef2113c7 100644
--- a/ci/incremental/setup_conda_environment.cmd
+++ b/ci/incremental/setup_conda_environment.cmd
@@ -11,11 +11,10 @@ call deactivate
 @rem Display root environment (for debugging)
 conda list
 @rem Clean up any left-over from a previous build
-conda remove --all -q -y -n %CONDA_ENV%
+conda remove --all -q -y -n pandas-dev
 @rem Scipy, CFFI, jinja2 and IPython are optional dependencies, but exercised in the test suite
-conda env create -n %CONDA_ENV% --file=ci\deps\azure-windows-%CONDA_PY%.yaml
+conda env create -n pandas-dev --file=ci\deps\azure-windows-%CONDA_PY%.yaml
 
-call activate %CONDA_ENV%
 conda list
 
 if %errorlevel% neq 0 exit /b %errorlevel%
diff --git a/ci/incremental/setup_conda_environment.sh b/ci/incremental/setup_conda_environment.sh
index f3ac99d5e7c5a..00d9fd08171a4 100755
--- a/ci/incremental/setup_conda_environment.sh
+++ b/ci/incremental/setup_conda_environment.sh
@@ -5,6 +5,7 @@ set -v -e
 CONDA_INSTALL="conda install -q -y"
 PIP_INSTALL="pip install -q"
 
+
 # Deactivate any environment
 source deactivate
 # Display root environment (for debugging)
@@ -12,16 +13,12 @@ conda list
 # Clean up any left-over from a previous build
 # (note workaround for https://github.com/conda/conda/issues/2679:
 #  `conda env remove` issue)
-conda remove --all -q -y -n $CONDA_ENV
+conda remove --all -q -y -n pandas-dev
 
 echo
 echo "[create env]"
-time conda env create -q -n "${CONDA_ENV}" --file="${ENV_FILE}" || exit 1
+time conda env create -q -n pandas-dev --file="${ENV_FILE}" || exit 1
 
-# Activate first
-set +v
-source activate $CONDA_ENV
-set -v
 
 # remove any installed pandas package
 # w/o removing anything else
diff --git a/ci/script_single.sh b/ci/script_single.sh
index ea0d48bc2da8a..b57b643290c73 100755
--- a/ci/script_single.sh
+++ b/ci/script_single.sh
@@ -2,7 +2,6 @@
 
 echo "[script_single]"
 
-source activate pandas
 
 if [ -n "$LOCALE_OVERRIDE" ]; then
     echo "Setting LC_ALL and LANG to $LOCALE_OVERRIDE"
diff --git a/ci/upload_coverage.sh b/ci/upload_coverage.sh
index a7ef2fa908079..88aca20590505 100755
--- a/ci/upload_coverage.sh
+++ b/ci/upload_coverage.sh
@@ -5,7 +5,6 @@ if [ -z "$COVERAGE" ]; then
    exit 0
 fi
 
-source activate pandas
 
 echo "uploading coverage"
 bash <(curl -s https://codecov.io/bash) -Z -c -F single -f /tmp/cov-single.xml


Code review comments:
diff_hunk: @@ -35,21 +35,24 @@ jobs:
         ci/incremental/install_miniconda.sh
         export PATH=$HOME/miniconda3/bin:$PATH
         echo "Setting up Conda environment"
+        source activate pandas-dev
path: ci/azure/linux.yml
text: @user1:
if I'm not wrong, the environment is created in the next file, and it doesn't exist here, so it can't be activated (and it's not needed)

diff_hunk: @@ -11,11 +11,10 @@ call deactivate
 @rem Display root environment (for debugging)
 conda list
 @rem Clean up any left-over from a previous build
-conda remove --all -q -y -n %CONDA_ENV%
+conda remove --all -q -y -n pandas-dev
 @rem Scipy, CFFI, jinja2 and IPython are optional dependencies, but exercised in the test suite
-conda env create -n %CONDA_ENV% --file=ci\deps\azure-windows-%CONDA_PY%.yaml
+conda env create -n pandas-dev --file=ci\deps\azure-windows-%CONDA_PY%.yaml
path: ci/incremental/setup_conda_environment.cmd
text: @user1:
you don't need the `-n pandas-dev`, the name of the environment is already defined in the yaml file

diff_hunk: @@ -5,23 +5,20 @@ set -v -e
 CONDA_INSTALL="conda install -q -y"
 PIP_INSTALL="pip install -q"
 
+
 # Deactivate any environment
 source deactivate
 # Display root environment (for debugging)
 conda list
 # Clean up any left-over from a previous build
 # (note workaround for https://github.com/conda/conda/issues/2679:
 #  `conda env remove` issue)
-conda remove --all -q -y -n $CONDA_ENV
+conda remove --all -q -y -n pandas-dev
 
 echo
 echo "[create env]"
-time conda env create -q -n "${CONDA_ENV}" --file="${ENV_FILE}" || exit 1
+time conda env create -q -n pandas-dev --file="${ENV_FILE}" || exit 1
 
-# Activate first
-set +v
-source activate $CONDA_ENV
path: ci/incremental/setup_conda_environment.sh
text: @user1:
This one can't be removed, as the rest of the next step is to remove pandas from the environment, so it needs to be active

diff_hunk: @@ -5,23 +5,20 @@ set -v -e
 CONDA_INSTALL="conda install -q -y"
 PIP_INSTALL="pip install -q"
 
+
 # Deactivate any environment
 source deactivate
 # Display root environment (for debugging)
 conda list
 # Clean up any left-over from a previous build
 # (note workaround for https://github.com/conda/conda/issues/2679:
 #  `conda env remove` issue)
-conda remove --all -q -y -n $CONDA_ENV
+conda remove --all -q -y -n pandas-dev
 
 echo
 echo "[create env]"
-time conda env create -q -n "${CONDA_ENV}" --file="${ENV_FILE}" || exit 1
+time conda env create -q -n pandas-dev --file="${ENV_FILE}" || exit 1
path: ci/incremental/setup_conda_environment.sh
text: @user1:
same about `-n pandas-dev` not needed