opened
shealtielcanon
username_0
Hi @username_1 

Trying to parallelize a function with multiple arguments. The case here is that the code runs into an IOError when main() is called.

The function evaluateAll() is to be parallelized.

def evaluateAll(curriculum, [finalGeneration[index]], status, searchpool, preferences, yrlvl, sem, dist, units):
	#some more functions here
return list

def main():
	with pymp.Parallel(5) as q:
		for index in q.range(5):
			evaluateAll(curriculum, [finalGeneration[index]], status, searchpool, preferences, yrlvl, sem, dist, units)
Initializing Parallel context runs into IOError
issue
created
classner
username_1
Hi @username_0,

pymp does not do anything special about functions called in parallel sections. This should just work fine. I do not see how your declaration of `evaluateAll` works, however, but I assume that's just a quickly assembled example.

Can you provide a minimum breaking example that I could use to reproduce the problem? You could use this as a starting point: 

```
#!/usr/bin/env python
import pymp


def evaluateAll(p1, p2):
    return p1


def main():
    with pymp.Parallel(5) as p:
        for index in p.range(5):
            evaluateAll(index, 3)
    return "res"


if __name__ == "__main__":
    main()
```

This still runs just fine. Otherwise, let me know whether you solved the problem in the meantime.
comment
created
shealtielcanon
username_0
Thank you for the response! @username_1 

evaluateAll is defined as:

```python
def evaluateAll(curriculum, loads, scholasticStatus, subjectsNotTaken, orientation, yearLevel, semester, dist, units):
	a = evaluateConflict(loads)
	b = evaluateUnits(loads, scholasticStatus, units)
	c = evaluateRequisites(loads, subjectsNotTaken)
	d = evaluateDayOrient(loads, orientation)
	e = evaluateCurriculum(loads, yearLevel, semester, curriculum)
	f = evaluateDist(loads, dist)

	a_new = [x * 0.3 for x in a]
	b_new = [x * 0.2 for x in b]
	c_new = [x * 0.2 for x in c]
	d_new = [x * 0.05 for x in d]
	e_new = [x * 0.2 for x in e]
	f_new = [x * 0.05 for x in f]

	container = zip(a_new, b_new, c_new, d_new, e_new, f_new)
	fitness_eval = [(x[0] + x[1] + x[2] + x[3] + x[4] + x[5]) for x in container]

	return [fitness_eval, a[0], b[0], c[0], d[0], e[0], f[0]]
```
There are also functions called in evaluateAll and the gist is to parallelized evaluateAll called in the main function.
comment
created
classner
username_1
Ok! I don't see any problem in the definition of evaluateAll. One thing that could cause trouble is the return value of evaluateAll, which will not be available in the main thread (however, in your minimal example you were not using it).

Can you provide a minimal but fully executable code example with dummy values that is breaking as in your application case? Then I could probably give a more insightful response.
comment
created
shealtielcanon
username_0
I tried reproducing the function outside of the production server and apparently it works just fine. The error only occurs inside the production server with the gunicorn configuration. I am not sure whether my configuration has something to do with the error. I also tried instantiating the `Parallel` context on other parts of the application and it still yields the same error which led me to the hypothesis that maybe its my configuration that deters the `Parallel` function.
comment
created
classner
username_1
Hmm, that's actually helpful information. In that case I assume the problem is not within pymp. pymp works by forking processes and I assume that gunicorn prevents the fork operation from working (the forking is done when entering a parallel section, so exactly where you notice the error). To find a solution, I would look into the gunicorn docs and maybe open an issue there if the forking behavior is not documented.
comment
created
shealtielcanon
username_0
Tweaked my `gunicorn` and it appears to have solved the error. The `Parallel` section is running as it is intended.

Thank you for your help! @username_1
comment
created
classner
username_1
Glad to hear that! Good luck with your project!
comment
closed
classner
username_1
issue