Repository: carpentries-incubator/python-intermediate-development
Pull request #436: Add explanations around installing our project locally
Author: login: bielsnohr
Created: 2025-02-25T15:35:13Z | Merged: 2025-03-07T17:08:49Z

As explained in #278, it is often a point of friction that learners don't have the local project installed in their virtual environment. Even though a lot of this can be alleviated by using `python -m <command>`, there are still deficits in that this command *must* be executed from the top level of the project where the `inflammation/` source directory is located.

As I also mention in that issue, it is standard practice to locally install the package you are working on and do so with an editable install. At the level of this course, I think this is an essential practice to impart.

Closes #278 

Please also see https://github.com/carpentries-incubator/python-intermediate-inflammation/pull/41 for the relevant change needed for the learner repo.

Linked issue #278: Add minimum pyproject.toml so project repo can be installed in learner environments
As mentioned in #268, the lack of a way to easily install the project repo in a virtual environment makes it quite difficult to navigate some aspects of the course, particularly exercising parts of the IDE and testing. It is worth noting, that almost any Python project I have ever worked on requires an editable install of the package. I recommend we return to doing the same at the beginning of the lesson.

On the practical side, this requires a fairly rudimentary `pyproject.toml` to get running:

```toml
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "python-intermediate-inflammation"
version = "0.0.0"
requires-python = ">=3.8"

[tool.setuptools]
packages = ["inflammation"]
```

This would be bundled into the initial learner project. Only thing that would need to be added is an install command put somewhere in the course (probably before the testing lesson):

```bash
pip install -e .
```

Linked issue comments: user: login: steve-crouch
created_at: 2025-02-25T10:26:26Z
body: Proposed solution would simplify the running of this episode, would be a forward reference? Add a callout that addresses this, and mentions the advantages of it.
author_association: COLLABORATOR
user: login: bielsnohr
created_at: 2025-02-25T10:30:55Z
body: Two things to add from maintainers' meeting:

1. Check whether pip has been introduced before this
2. As mentioned by @steve-crouch above, we could add a collapsable callout that forward references the Poetry section about what `pyproject.toml` does (just so we aren't smoothing things over too much).
author_association: COLLABORATOR

Diff:
diff --git a/episodes/12-virtual-environments.md b/episodes/12-virtual-environments.md
index afa971423..da7c2357c 100644
--- a/episodes/12-virtual-environments.md
+++ b/episodes/12-virtual-environments.md
@@ -483,6 +483,67 @@ six             1.16.0
 To uninstall a package installed in the virtual environment do: `python3 -m pip uninstall <package-name>`.
 You can also supply a list of packages to uninstall at the same time.
 
+### Installing Our Local Project as a Package Using `pip`
+
+Often when working on a Python project, the project itself will be a Python package (like `numpy` or `matplotlib` above) or at the very least it might be useful to treat it like a package.
+Said another way, it is usually the case we want a convenient way to call the Python code we are writing from another location, and making this code accessible as a package is the best way to do this.
+We will save the details of Python packaging for [a future episode](43-software-release.md), and for the meantime we can use the minimal package setup that our project already comes with, which is contained in the `pyproject.toml` file.
+Once again, we can use `pip` to install our local package:
+
+```bash
+python3 -m pip install --editable .
+```
+
+:::::::::::::::::::::::::::::::::::::::::  spoiler
+
+### This command fails for me
+
+If your `pip` installation is older than version 21.3, then this command will probably fail for you.
+This is because these older versions of `pip` do not support `pyproject.toml` as the package metadata.
+Given these versions of `pip` are now over 4 years old, we strongly recommend that you update `pip` if you can:
+
+```bash
+python3 -m pip install --upgrade pip
+```
+
+:::::::::::::::::::::::::::::::::::::::::
+
+This is similar syntax to above, with two important differences:
+
+1. The `--editable` or `-e` flag indicates that the package we are specifying should be an "editable" install.
+   An "editable" install is one that allows the package in our environment to change dynamically based on source code locally.
+   This is very convenient when we are developing the package because we can instantly see changes when we call the code from within our virtual environment, rather than having to install the local package again to get the updates.
+2. The argument `'.'` indicates that the package we want to install is located in the current directory.
+   The `pyproject.toml` file located in this directory then handles the rest.
+
+
+
+If we reissue the `pip list` command we should now see our local package with the name `python-intermediate-inflammation` in the output:
+
+```output
+Package                          Version     Editable project location
+-------------------------------- ----------- ----------------------------------------------------------------------------------------------
+contourpy                        1.3.1
+cycler                           0.12.1
+exceptiongroup                   1.2.2
+fonttools                        4.56.0
+iniconfig                        2.0.0
+kiwisolver                       1.4.8
+matplotlib                       3.10.0
+numpy                            2.2.3
+packaging                        24.2
+pillow                           11.1.0
+pip                              22.0.2
+pluggy                           1.5.0
+pyparsing                        3.2.1
+pytest                           8.3.4
+python-dateutil                  2.9.0.post0
+python-intermediate-inflammation 0.0.0       /path/to/your/project/directory/python-intermediate-inflammation
+setuptools                       59.6.0
+six                              1.17.0
+tomli                            2.2.1
+```
+
 ### Exporting/Importing Virtual Environments Using `pip`
 
 You are collaborating on a project with a team so, naturally,
@@ -491,12 +552,11 @@ so they can easily 'clone' your software project with all of its dependencies
 and everyone can replicate equivalent virtual environments on their machines.
 `pip` has a handy way of exporting, saving and sharing virtual environments.
 
-To export your active environment -
-use `python3 -m pip freeze` command to produce a list of packages installed in the virtual environment.
+To export your active environment use the `python3 -m pip freeze --exclude-editable` command to produce a list of packages installed in the virtual environment.
 A common convention is to put this list in a `requirements.txt` file:
 
 ```bash
-(venv) $ python3 -m pip freeze > requirements.txt
+(venv) $ python3 -m pip freeze --exclude-editable > requirements.txt
 (venv) $ cat requirements.txt
 ```
 
@@ -518,6 +578,7 @@ The first of the above commands will create a `requirements.txt` file in your cu
 Yours may look a little different,
 depending on the version of the packages you have installed,
 as well as any differences in the packages that they themselves use.
+Also, we need to use the `--exclude-editable` command so that our local package is not included in the output, otherwise pip will try to pull from a specific commit at the time we made the editable install, which is not what we want.
 
 The `requirements.txt` file can then be committed to a version control system
 (we will see how to do this using Git in one of the following episodes)
@@ -526,10 +587,10 @@ They can then replicate your environment
 and install all the necessary packages from the project root as follows:
 
 ```bash
-(venv) $ python3 -m pip install -r requirements.txt
+(venv) $ python3 -m pip install -r requirements.txt --editable .
 ```
 
-As your project grows - you may need to update your environment for a variety of reasons.
+As your project grows you may need to update your environment for a variety of reasons.
 For example, one of your project's dependencies has just released a new version
 (dependency version number update),
 you need an additional package for data analysis (adding a new dependency)
@@ -604,7 +665,7 @@ to customize the command line.
 - Use `venv` to create and manage Python virtual environments.
 - Use `pip` to install and manage Python external (third-party) libraries.
 - `pip` allows you to declare all dependencies for a project in a separate file (by convention called `requirements.txt`) which can be shared with collaborators/users and used to replicate a virtual environment.
-- Use `python3 -m pip freeze > requirements.txt` to take snapshot of your project's dependencies.
+- Use `python3 -m pip freeze --exclude-editable > requirements.txt` to take snapshot of your project's dependencies.
 - Use `python3 -m pip install -r requirements.txt` to replicate someone else's virtual environment on your machine from the `requirements.txt` file.
 
 ::::::::::::::::::::::::::::::::::::::::::::::::::
diff --git a/episodes/13-ides.md b/episodes/13-ides.md
index 5324931f3..3feb6097d 100644
--- a/episodes/13-ides.md
+++ b/episodes/13-ides.md
@@ -203,7 +203,7 @@ you may remember that we can get the list of packages in the current virtual env
 using `pip`:
 
 ```bash
-(venv) $ python3 -m pip list
+(venv) $ python3 -m pip list --exclude-editable
 ```
 
 ```output
@@ -231,7 +231,7 @@ if we want to see only the list of packages that we installed,
 we can use the `python3 -m pip freeze` command instead:
 
 ```bash
-(venv) $ python3 -m pip freeze
+(venv) $ python3 -m pip freeze --exclude-editable
 ```
 
 ```output
@@ -250,13 +250,13 @@ six==1.16.0
 
 We see the `pip` package in `python3 -m pip list` but not in `python3 -m pip freeze` 
 as we did not install it using `pip`.
-Remember that we use `python3 -m pip freeze` to update our `requirements.txt` file,
+Remember that we use `python3 -m pip freeze --exclude-editable` to update our `requirements.txt` file,
 to keep a list of the packages our virtual environment includes.
 Python will not do this automatically;
 we have to manually update the file when our requirements change using:
 
 ```bash
-python3 -m pip freeze > requirements.txt
+python3 -m pip freeze --exclude-editable > requirements.txt
 ```
 
 If we want, we can also see the list of packages directly in the following subdirectory of `venv`:
@@ -399,7 +399,7 @@ six==1.16.0
 `pytest` is missing from `requirements.txt`. To add it, we need to update the file by repeating the command:
 
 ```bash
-(venv) $ python3 -m pip freeze > requirements.txt
+(venv) $ python3 -m pip freeze --exclude-editable > requirements.txt
 ```
 
 `pytest` is now present in `requirements.txt`:
diff --git a/episodes/16-verifying-code-style-linters.md b/episodes/16-verifying-code-style-linters.md
index 5a9ad6f8d..ae185c5f1 100644
--- a/episodes/16-verifying-code-style-linters.md
+++ b/episodes/16-verifying-code-style-linters.md
@@ -44,7 +44,7 @@ $ python3 -m pip install pylint
 We should also update our `requirements.txt` with this new addition:
 
 ```bash
-$ python3 -m pip freeze > requirements.txt
+$ python3 -m pip freeze --exclude-editable > requirements.txt
 ```
 
 Pylint is a command-line tool that can help our code in many ways:
diff --git a/episodes/21-automatically-testing-software.md b/episodes/21-automatically-testing-software.md
index 6a420e5f8..152ca9cc9 100644
--- a/episodes/21-automatically-testing-software.md
+++ b/episodes/21-automatically-testing-software.md
@@ -614,7 +614,7 @@ Since we have installed `pytest` to our environment,
 we should also regenerate our `requirements.txt`:
 
 ```bash
-$ python3 -m pip freeze > requirements.txt
+$ python3 -m pip freeze --exclude-editable > requirements.txt
 ```
 
 Finally, let us commit our new `test_models.py` file,
diff --git a/episodes/22-scaling-up-unit-testing.md b/episodes/22-scaling-up-unit-testing.md
index 0831600af..44e76ba85 100644
--- a/episodes/22-scaling-up-unit-testing.md
+++ b/episodes/22-scaling-up-unit-testing.md
@@ -213,7 +213,7 @@ Again, we should also update our `requirements.txt` file with our latest package
 which now also includes `pytest-cov`, and commit it:
 
 ```bash
-$ python3 -m pip freeze > requirements.txt
+$ python3 -m pip freeze --exclude-editable > requirements.txt
 $ cat requirements.txt
 ```
 
diff --git a/episodes/23-continuous-integration-automated-testing.md b/episodes/23-continuous-integration-automated-testing.md
index b412166db..4b167d02b 100644
--- a/episodes/23-continuous-integration-automated-testing.md
+++ b/episodes/23-continuous-integration-automated-testing.md
@@ -200,7 +200,7 @@ jobs:
     - name: Install Python dependencies
       run: |
         python3 -m pip install --upgrade pip
-        python3 -m pip install -r requirements.txt
+        python3 -m pip install -r requirements.txt .
 
     - name: Test with PyTest
       run: |
@@ -244,7 +244,7 @@ Each of these steps are:
   In order to locally install our `inflammation` package
   it is good practice to upgrade the version of pip that is present first,
   then we use pip to install our package dependencies.
-  Once installed, we can use `python3 -m pip install -e .` as before to install our own package.
+  Notice that it is fine to omit the `--editable` flag in this case because the source code will be static and therefore we don't need this to be an "editable" install like when we are developing locally.
   We use `run` here to run theses commands in the CI shell environment
 - **Test with PyTest:** lastly, we run `python3 -m pytest`,
   with the same arguments we used manually before
@@ -383,10 +383,10 @@ jobs:
     - name: Install Python dependencies
       run: |
         python3 -m pip install --upgrade pip
-        python3 -m pip install -r requirements.txt
+        python3 -m pip install -r requirements.txt .
     - name: Test with PyTest
       run: |
-        python3 -m pytest --cov=catchment.models tests/test_models.py
+        python3 -m pytest --cov=inflammation.models tests/test_models.py
 ```
 
 The `{{ }}` are used
diff --git a/slides/section_1_setting_up_environment.md b/slides/section_1_setting_up_environment.md
index df6a3fecc..d4ba7fc88 100644
--- a/slides/section_1_setting_up_environment.md
+++ b/slides/section_1_setting_up_environment.md
@@ -341,9 +341,9 @@ Read through and follow along until the end of the episode page.
 rm -r venv/
 python3 -m venv venv
 source venv/bin/activate
-pip install <your_dependencies> 
+pip install <list_your_dependencies>
 # or
-pip install -r requirements.txt  # great reason to have this file
+pip install -r requirements.txt # great reason to have this file
 
 ```
 <!-- #endregion -->
diff --git a/slides/section_4_collaborative_soft_dev.md b/slides/section_4_collaborative_soft_dev.md
index 92c8d8323..d7678f8a6 100644
--- a/slides/section_4_collaborative_soft_dev.md
+++ b/slides/section_4_collaborative_soft_dev.md
@@ -305,7 +305,7 @@ poetry --version  # check we have access to the poetry executable
   git clone <our_repo>
   python -m venv venv
   . venv/bin/activate
-  pip install -r requirements.txt
+  pip install -r requirements.txt --editable .
   python inflammation-analysis.py
   ...
   ```


Reviews:
user: login: svenvanderburg
body: Great addition! I think this is a sufficient amount of explanation, and I think it is very clear.

I made a few suggestions for where I think we should change the specific pip commands.
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-03T09:19:53Z
user: login: svenvanderburg
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-03T09:21:28Z
user: login: bielsnohr
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-03T21:00:38Z
user: login: bielsnohr
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-03T21:05:08Z
user: login: bielsnohr
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-03T21:07:42Z

Review comments:
diff_hunk: @@ -526,10 +571,10 @@ They can then replicate your environment
 and install all the necessary packages from the project root as follows:
 
 ```bash
-(venv) $ python3 -m pip install -r requirements.txt
+(venv) $ python3 -m pip install -r requirements.txt --editable .
path: episodes/12-virtual-environments.md
user: login: svenvanderburg
body: ```suggestion
(venv) $ python3 -m pip install -r requirements.txt
```
Right?
created_at: 2025-03-03T09:15:27Z
author_association: COLLABORATOR
diff_hunk: @@ -341,9 +341,9 @@ Read through and follow along until the end of the episode page.
 rm -r venv/
 python3 -m venv venv
 source venv/bin/activate
-pip install <your_dependencies> 
+pip install --editable . <list_your_dependencies>
path: slides/section_1_setting_up_environment.md
user: login: svenvanderburg
body: ```suggestion
pip install <list_your_dependencies>
```
I think this should stay what it was right?
created_at: 2025-03-03T09:17:46Z
author_association: COLLABORATOR
diff_hunk: @@ -341,9 +341,9 @@ Read through and follow along until the end of the episode page.
 rm -r venv/
 python3 -m venv venv
 source venv/bin/activate
-pip install <your_dependencies> 
+pip install --editable . <list_your_dependencies>
 # or
-pip install -r requirements.txt  # great reason to have this file
+pip install -r requirements.txt --editable .  # great reason to have this file
path: slides/section_1_setting_up_environment.md
user: login: svenvanderburg
body: ```suggestion
pip install -r requirements.txt # great reason to have this file
```
created_at: 2025-03-03T09:18:29Z
author_association: COLLABORATOR
diff_hunk: @@ -305,7 +305,7 @@ poetry --version  # check we have access to the poetry executable
   git clone <our_repo>
   python -m venv venv
   . venv/bin/activate
-  pip install -r requirements.txt
+  pip install -r requirements.txt --editable .
path: slides/section_4_collaborative_soft_dev.md
user: login: svenvanderburg
body: ```suggestion
  pip install -r requirements.txt
```
created_at: 2025-03-03T09:18:43Z
author_association: COLLABORATOR
diff_hunk: @@ -526,10 +571,10 @@ They can then replicate your environment
 and install all the necessary packages from the project root as follows:
 
 ```bash
-(venv) $ python3 -m pip install -r requirements.txt
+(venv) $ python3 -m pip install -r requirements.txt --editable .
path: episodes/12-virtual-environments.md
user: login: svenvanderburg
body: I think here we should just install the depencies as normal, and don't make all depencies editable installs.
created_at: 2025-03-03T09:21:28Z
author_association: COLLABORATOR
diff_hunk: @@ -526,10 +571,10 @@ They can then replicate your environment
 and install all the necessary packages from the project root as follows:
 
 ```bash
-(venv) $ python3 -m pip install -r requirements.txt
+(venv) $ python3 -m pip install -r requirements.txt --editable .
path: episodes/12-virtual-environments.md
user: login: bielsnohr
body: This doesn't make all of the dependencies editable, only those after the `--editable` flag. Everything within `requirements.txt` will be installed "normally" as long as it isn't pre-pended by `-e`. See https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-e
created_at: 2025-03-03T21:00:37Z
author_association: COLLABORATOR
diff_hunk: @@ -341,9 +341,9 @@ Read through and follow along until the end of the episode page.
 rm -r venv/
 python3 -m venv venv
 source venv/bin/activate
-pip install <your_dependencies> 
+pip install --editable . <list_your_dependencies>
path: slides/section_1_setting_up_environment.md
user: login: bielsnohr
body: Yes, good catch! We don't introduce the `--editable` install until the next episode.
created_at: 2025-03-03T21:05:08Z
author_association: COLLABORATOR
diff_hunk: @@ -305,7 +305,7 @@ poetry --version  # check we have access to the poetry executable
   git clone <our_repo>
   python -m venv venv
   . venv/bin/activate
-  pip install -r requirements.txt
+  pip install -r requirements.txt --editable .
path: slides/section_4_collaborative_soft_dev.md
user: login: bielsnohr
body: I think in this case it should stay here. After episode `12-virtual-environments` all installations of our project should be done in editable mode.
created_at: 2025-03-03T21:07:42Z
author_association: COLLABORATOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-02-25T15:35:38Z
body: ## Thank you!

Thank you for your pull request :smiley:

:robot: This automated message can help you check the rendered files in your submission for clarity. If you have any questions, please feel free to [open an issue in {sandpaper}](https://github.com/carpentries/sandpaper/issues/new).

If you have files that automatically render output (e.g. R Markdown), then you should check for the following:

 - :dart: correct output
 - :framed_picture: correct figures
 - :question: new warnings
 - :bangbang: new errors

### Rendered Changes

:mag: Inspect the changes: https://github.com/carpentries-incubator/python-intermediate-development/compare/md-outputs..md-outputs-PR-436


The following changes were observed in the _rendered_ markdown documents:

```
 12-virtual-environments.md                     | 73 +++++++++++++++++++++++---
 13-ides.md                                     | 10 ++--
 16-verifying-code-style-linters.md             |  2 +-
 21-automatically-testing-software.md           |  2 +-
 22-scaling-up-unit-testing.md                  |  2 +-
 23-continuous-integration-automated-testing.md |  8 +--
 md5sum.txt                                     | 16 +++---
 section_1_setting_up_environment.md            |  4 +-
 section_4_collaborative_soft_dev.md            |  2 +-
 9 files changed, 90 insertions(+), 29 deletions(-)
```

<details>
<summary>What does this mean?</summary>

If you have source files that require output and figures to be generated (e.g.  R Markdown), then it is important to make sure the generated figures and output are reproducible.

This output provides a way for you to inspect the output in a diff-friendly manner so that it's easy to see the changes that occur due to new software versions or randomisation.

</details>

:stopwatch: Updated at 2025-03-07 17:08:17 +0000

author_association: NONE
user: login: bielsnohr
created_at: 2025-03-03T21:17:48Z
body: Thanks for the review @svenvanderburg ! I've incorporated some of the changes, but declined some of the others. Requesting a quick re-review from you to make sure it all looks in order.
author_association: COLLABORATOR
user: login: bielsnohr
created_at: 2025-03-07T17:08:41Z
body: Okay, I feel like I have addressed everything raised here, and I would really like this to be in the course content for my run of the course starting on Monday. I'm going to go ahead an merge. Any errors will be caught by my team as we are running this through in the coming weeks.
author_association: COLLABORATOR
user: login: svenvanderburg
created_at: 2025-03-10T11:01:57Z
body: @bielsnohr totally fine for me! Good luck with the course 👍
author_association: COLLABORATOR