Repository: Parsl/parsl

Issue:
Outreachy: getting familiar with the development environment
This issue is targeted at potential applications for Parsl's participation in Outreachy. The tasks in this issue can be performed by any number of applicants. Talk to @benclifford on Slack if you are interested.

The point of the tasks in this issue are to walk you through getting a development environment set up, making a trivial change to the code, seeing how our tooling tests things, deliberately introducing a problem, and then fixing that problem. By the end you should have a trivial 1-byte change to submit to parsl as a PR, to try out the PR process.

There are lots of steps because I have tried to make each step small.

If you get stuck or don't understand any of these steps - you can ask @benclifford on Slack or write publicly in the #parsl-hackers channel on  Slack.

1. Get a Python environment - parsl is supported on Python versions 3.8 - 3.12.
2. Sign up for a GitHub account
3. Go to the parsl repository at https://github.com/Parsl/parsl and near the top right, click "Forks / create a new fork". This will make a fork of parsl in your own github account.
5. Install parsl - follow the instructions in the "For developers" section of the Parsl README at https://github.com/Parsl/parsl - but when asked to run the command `git clone https://github.com/parsl/parsl`, instead use the address of your fork created in the previous step.
6. You can run some local tests by running `make local_thread_test`, and some more advanced tests with `make htex_local_alternate_test`
7. You can test our code linting by running `make flake8' which checks all of our Python code to see if it matches the Python style rules in [PEP-8](https://peps.python.org/pep-0008/)
8. We have configuration to let us break the style rules sometimes. This lives in the file called `.flake8`. You can open that file and make a change to move us towards proper PEP-8 compliance: find the line that says `max-line-length = 158` and make it 1 smaller. (PEP-8 says the max line length should be 80... but we can't change it all at once)
9. Run `make flake8` again - you should see that you can an error: flake8 should have found a line that is now too long.
10. Edit that line - you'll have to figure out how to make it shorter, perhaps by splitting it into two valid lines of Python
11. Run `make flake8` again and you should see that flake8 passes again
12. Now you can make a git commit: type `git commit -a -m "Reduce flake8 max-line-length`. Don't worry if you don't understand what git is doing: you can learn more about that during the internship.
13. Now push that commit up to github: type `git push`
14. Visit your repository on github in your web browser - you should see your commit there now.
15. Click "Pull request" to make a pull request to the main Parsl repository.
16. Fill out the template - you don't need to write too much
17. Sit back and wait for someone to review your pull request - that might happen within minutes, or it might take many hours, depending on who is working/awake/etc. Don't panic if you haven't heard after a few hours!
18. You might get some feedback and need to make some changes - the reviewer will help you with that
19. or... they might merge your change, in which case Congratulations! you've made a tiny contribution to Parsl, and our code base is now a tiny bit cleaner!


Discussion:
My application got approved in outreachy, I will take this and create a pr
@Harichandra-Prasath ok great! This specific issue #3105 isn't limited to one applicant: it's a basic exercise for everyone to get a development environment going, so other people reading this shoudn't be put off trying this out too.
I am very interested in this project, sir. May I know how to get started and do quality contributions. 
@Harichandra-Prasath work your way through this issue #3105 and get a pull request merged from a working dev environment, to get started - when that is all working, we can pick out some other issues that are good for you to work on.

@benclifford I am really interested in this project. I hope to learn and contribute to the project. 
Hello @benclifford !
### Could you spend a few minutes of your precious time reviewing #3128 ?

Thanks in advance!
Hey @benclifford please review #3126 . Thank you :).
Hello @benclifford please take a look!
Hi @benclifford I am an outreachy applicant and I am interested in contributing to this project. Kindly review my PR #3130  Thank you.
Hey @benclifford 

My initial application to outreachy got accepted and I'm interested to make contributions in this project. 
Please review:  **#3131**

Looking forward to your feedback. Thanks!
hi @benclifford, i just got accepted into the contribution phase of outreachy 2024, and i just created a PR on 

https://github.com/Parsl/parsl/pull/3144

pls review and merge. Thanks
> hi @benclifford, i just got accepted into the contribution phase of outreachy 2024, and i just created a PR on
> 
> #3144
> 
> pls review and merge. Thanks

thanks @rmeis06 for the insight, i have reviewed and edited my task and PR.  @benclifford pls review my PR https://github.com/Parsl/parsl/pull/3144 thanks
@benclifford  I have gone back to redo my work and i believe i did a much better job this time, pls review and merge. thanks
here is the link to my PR: https://github.com/Parsl/parsl/pull/3172
hi @benclifford  i have update the PR as requested, pls review and merge:  https://github.com/Parsl/parsl/pull/3176
Hello @benclifford i've done this task of reducing flake8 max-line length and open an PR [#3178](https://github.com/Parsl/parsl/pull/3178)

Apart from flake8 length, it also shows changes of `Assert fail` in the PR as well. So just ignore it for a while
while setting up this project I encountered some issues. I followed the instructions here and also on the readme, below are some of the issues and how I resolved them with the help of @benclifford 

1. when I ran the `make deps`  instruction from the readme file for Build and Test

**Issue:**  `ERROR: failed building wheel for mpi4py`

**Likely reason for the issues:** mpich is not installed for Linux users

**Solution:** `sudo apt install mpich` for Ubuntu users

2. when I ran the command on #3105 No. 5

**Issues:** 
```shell
pytest parsl/tests/ -k "not cleannet" --config parsl/tests/configs/local_threads.py --random-order --durations 10
/usr/bin/bash: line 1: pytest: command not found
make: *** [Makefile:52: local_thread_test] Error 127
``` 
**Likely reason for the issues:** The `make deps` didn't execute successfully, possibly because of my internet connection

**Solution:** I tried to rerun the `make deps` command again, this time it executed successfully.

then I was able to run the `make local_thread_test`  in #3105 No. 5 again and it worked

3. when I ran the build and test command it generated some random files and folders which are unnecessary.

**Issues:** These randomly generated files and folders were added to my commit when I tried to push because I ran the 
`git add . `  command

**Solution:** I ran the `git commit -a` command instead of using the `git add .` command and it will only commit files that were already there, not any new stuff

4. From the readme on the git repo the part that says `python3 setup.py install` in step 3 didn't work for me 

**Issue:** 
```shell
/home/work/Open-Source/parsl-fork/.venv/lib/python3.10/site-packages/setuptools/dist.py:493: UserWarning: Normalizing '1.3.0-dev' to '1.3.0.dev0'
  warnings.warn(tmpl.format(**locals()))
running install
/home/work/Open-Source/parsl-fork/.venv/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/home/work/Open-Source/parsl-fork/.venv/lib/python3.10/site-packages/setuptools/command/easy_install.py:158: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
error: [Errno 13] Permission denied
``` 

**Likely reason for the issues:**  I ran this command with sudo `make virtualenv` 

**Solution:**  I deleted the .venv folder by running  `sudo rm -r .venv`, also deleted the parsl.egg-info folder `sudo rm -r parsl.egg-info` and rerun the  `python3 setup.py install`  command


### NOTE:

if you are running a Windows OS on your PC, you might need to install WSL, this gives you an Ubuntu experience on your window machine

Read about how to install [WSL](https://learn.microsoft.com/en-us/windows/wsl/install) on your PC https://learn.microsoft.com/en-us/windows/wsl/install
> while setting up this project I encountered some issues. I followed the instructions here and also on the readme, below are some of the issues and how I resolved them with the help of @benclifford
> 
> 1. when I ran the `make deps`  instruction from the readme file for Build and Test
> 
> **Issue:** `ERROR: failed building wheel for mpi4py`
> 
> **Likely reason for the issues:** mpich is not installed for Linux users
> 
> **Solution:** `sudo apt install mpich` for Ubuntu users
> 
> 2. when I ran the command on [Outreachy: getting familiar with the development environment #3105](https://github.com/Parsl/parsl/issues/3105) No. 5
> 
> **Issues:**
> 
> ```shell
> pytest parsl/tests/ -k "not cleannet" --config parsl/tests/configs/local_threads.py --random-order --durations 10
> /usr/bin/bash: line 1: pytest: command not found
> make: *** [Makefile:52: local_thread_test] Error 127
> ```
> 
> **Likely reason for the issues:** The `make deps` didn't execute successfully, possibly because of my internet connection
> 
> **Solution:** I tried to rerun the `make deps` command again, this time it executed successfully.
> 
> then I was able to run the `make local_thread_test` in #3105 No. 5 again and it worked
> 
> 3. when I ran the build and test command it generated some random files and folders which are unnecessary.
> 
> **Issues:** These randomly generated files and folders were added to my commit when I tried to push because I ran the `git add . ` command
> 
> **Solution:** I ran the `git commit -a` command instead of using the `git add .` command and it will only commit files that were already there, not any new stuff
> 
> 4. From the readme on the git repo the part that says `python3 setup.py install` in step 3 didn't work for me
> 
> **Issue:**
> 
> ```shell
> /home/work/Open-Source/parsl-fork/.venv/lib/python3.10/site-packages/setuptools/dist.py:493: UserWarning: Normalizing '1.3.0-dev' to '1.3.0.dev0'
>   warnings.warn(tmpl.format(**locals()))
> running install
> /home/work/Open-Source/parsl-fork/.venv/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
>   warnings.warn(
> /home/work/Open-Source/parsl-fork/.venv/lib/python3.10/site-packages/setuptools/command/easy_install.py:158: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
>   warnings.warn(
> running bdist_egg
> running egg_info
> error: [Errno 13] Permission denied
> ```
> 
> **Likely reason for the issues:** I ran this command with sudo `make virtualenv`
> 
> **Solution:** I deleted the .venv folder by running `sudo rm -r .venv`, also deleted the parsl.egg-info folder `sudo rm -r parsl.egg-info` and rerun the `python3 setup.py install` command
> 
> ### NOTE:
> if you are running a Windows OS on your PC, you might need to install WSL, this gives you an Ubuntu experience on your window machine
> 
> Read about how to install [WSL](https://learn.microsoft.com/en-us/windows/wsl/install) on your PC https://learn.microsoft.com/en-us/windows/wsl/install

I hope this will be helpful to someone 
Hey  @benclifford 
I am an Outreachy applicant and I am interested in contributing to this project. 
Kindly review my PR -  [3138](https://github.com/Parsl/parsl/pull/3186) 
Thank you.
Hi @benclifford  please review my PR #3169 I also updated the readme.
While setting up on macos, I encoutered similar issues @OfficialOzioma faced.

When i ran `make deps` ,  i got these errors:  **ERROR: Failed building wheel for mpi4py
Failed to build mpi4py
ERROR: Could not build wheels for mpi4py, which is required to install pyproject.toml-based projects 
**
**RESOLUTION**
I ran these codes one after the other
`brew install open-mpi`
`pip3 install mpi4py`

When also running the `make test` command, I got the errors below
**process_worker_pool.py:643: error: Module has no attribute "sched_getaffinity"  [attr-defined]
parsl/executors/high_throughput/process_worker_pool.py:683: error: Module has no attribute "sched_setaffinity"  [attr-defined]
parsl/tests/test_python_apps/test_lifted.py:28: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
parsl/tests/test_python_apps/test_lifted.py:29: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
**
**RESOLUTION** 
I ran the smaller test suite `make config_local_test`

Also, if you get and error *flake8 not found* when running `make test`, use `brew` or `macports` depending on your package, manager of choice to install flake8 then re-run that same step 


@kemsguy7 ok, that sched_affinity message sounds like a cross-platform problem where we have tested things right on OS X - I'd be interested if you could paste the whole command and the whole output for that `make test` into a new issue, along with the version of your operating system, so that we can track it as a separate bug.
@kemsguy7 once you've done `brew install open-mpi`, then I think you should be able to run `make deps` again instead of separately installing mpi4py and flake8 (for example).
@SobiaAman asked elsewhere:

>  @benclifford is it okay that multiple contributors are doing the same thing and making the PR Or it should be done by only one contributor that comes first. Kindly clear that, i'm little bit confused in it. Thanks

Its fine for as many people try the more simple issues as they want - but in the end, we will only merge one PR, the first PR that is properly ready with all mentor/reviewer comments addressed. So if you want to try an issue to learn about it, even if someone else is already working on it, you are welcome to try it and get mentor/reviewer feedback - but if you see someone else is doing it, it might be that their PR is what gets merged, not yours, depending on which PR is fully ready to be approved first.
> @kemsguy7 ok, that sched_affinity message sounds like a cross-platform problem where we have tested things right on OS X - I'd be interested if you could paste the whole command and the whole output for that `make test` into a new issue, along with the version of your operating system, so that we can track it as a separate bug.

Okay, i'll do that

Okay @benclifford 
Got your point, Thanks

> hi @benclifford i have update the PR as requested, pls review and merge: #3176

@benclifford you still haven't reviewed my PR : #3176 
Hello everyone,

Let's address the errors to facilitate the creation of your pull request.

## why this issue occuring ?
because you have not fetch the recent changes in your pc

## Important message
 You should always use a new branch for every different task
and that each pull request comes from a different branch

## How to fix it ?
first `git remote add upstream https://github.com/parsl/parsl`
This will adds a new remote repository to your Git configuration.

then say `git fetch upstream `  
you should see some branches are updating

the next step is to make a new branch by
`git checkout -b <branch name> upstream/master`

if you type `git show` you will see recent commits on the branch

the next step is to make change in your code and is you type `git diff` you wiil see your changes

then make commit by `git commit -a`
this will commit changes without explict staging them
 
now type `git show`  should see your commit

now push your changes by `git push origin HEAD `

Resolving patch:
diff --git a/parsl/dataflow/dflow.py b/parsl/dataflow/dflow.py
--- a/parsl/dataflow/dflow.py
+++ b/parsl/dataflow/dflow.py
@@ -722,7 +722,10 @@ def launch_task(self, task_record: TaskRecord) -> Future:
         self._send_task_log_info(task_record)
 
         if hasattr(exec_fu, "parsl_executor_task_id"):
-            logger.info(f"Parsl task {task_id} try {try_id} launched on executor {executor.label} with executor id {exec_fu.parsl_executor_task_id}")
+            logger.info(
+                 f"Parsl task {task_id} try {try_id} launched on executor {executor.label} "
+                 f"with executor id {exec_fu.parsl_executor_task_id}"
+            )
         else:
             logger.info(f"Parsl task {task_id} try {try_id} launched on executor {executor.label}")
 
diff --git a/parsl/executors/taskvine/executor.py b/parsl/executors/taskvine/executor.py
--- a/parsl/executors/taskvine/executor.py
+++ b/parsl/executors/taskvine/executor.py
@@ -228,7 +228,10 @@ def __create_data_and_logging_dirs(self):
         # factory logs go with manager logs regardless
         self.factory_config.scratch_dir = self.manager_config.vine_log_dir
         logger.debug(f"Function data directory: {self._function_data_dir}, log directory: {log_dir}")
-        logger.debug(f"TaskVine manager log directory: {self.manager_config.vine_log_dir}, factory log directory: {self.factory_config.scratch_dir}")
+        logger.debug(
+            f"TaskVine manager log directory: {self.manager_config.vine_log_dir}, "
+            f"factory log directory: {self.factory_config.scratch_dir}"
+        )
 
     def start(self):
         """Create submit process and collector thread to create, send, and


Test patch:
diff --git a/parsl/tests/configs/user_opts.py b/parsl/tests/configs/user_opts.py
--- a/parsl/tests/configs/user_opts.py
+++ b/parsl/tests/configs/user_opts.py
@@ -52,7 +52,11 @@
     #     'username': MIDWAY_USERNAME,
     #     'script_dir': '/scratch/midway2/{}/parsl_scripts'.format(MIDWAY_USERNAME),
     #     'scheduler_options': "",
-    #     'worker_init': 'cd /scratch/midway2/{}/parsl_scripts; module load Anaconda3/5.1.0; source activate parsl_testing;'.format(MIDWAY_USERNAME),
+    #     'worker_init': (
+    #         'cd /scratch/midway2/{}/parsl_scripts; '
+    #         'module load Anaconda3/5.1.0; '
+    #         'source activate parsl_testing;'
+    #      ).format(MIDWAY_USERNAME),
     # },
     # 'osg': {
     #     'username': OSG_USERNAME,