Parallel Cythonization for spawned Processes
State: closed

Originally thought this only affected Windows, but parallel Cythonization is also broken on Python 3.8 on macOS as they shifted from using fork to start sub-processes to using spawn

https://bugs.python.org/issue33725

This fix should make things work regardless of platform / python version


Labels:
name: Enhancement
name: Build
description: Library building on various platforms

Comments:
@Dr-Irv @topper-123 
@WillAyd The cython folks saw that Python 3.8 on macos issue here:  https://github.com/conda-forge/cython-feedstock/pull/59 (and maybe elsewhere)

Once you are merged, ping me, and I'll test on Windows.

I've tried this out on windows and it works for me. I can't however see if the build is running in parallel, maybe @Dr-Irv can comment on that?
I now learned how to fetch this pull request and tried it out.  Definitely goes parallel (I use -j 8 since I have 8 real cores).  BUT there is an issue that some of the compiles of the C code are not finished before the final "cl" step.  So that step fails, and I have to do `python setup.py build_ext --inplace -j 8` again and then it works.
What do you mean by "cl" step? Also is it failing during cythonize or during generation of the libraries?
When I try a clean build, here is the end of what is on the screen:
```
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Anaconda3\envs\pandas-dev\libs /LIBPATH:C:\Anaconda3\envs\pandas-dev\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" /EXPORT:PyInit_json build\temp.win-amd64-3.7\Release\pandas/_libs/src/ujson/python/ujson.obj build\temp.win-amd64-3.7\Release\pandas/_libs/src/ujson/python/objToJSON.obj build\temp.win-amd64-3.7\Release\pandas/_libs/src/ujson/python/JSONtoObj.obj build\temp.win-amd64-3.7\Release\pandas/_libs/src/ujson/lib/ultrajsonenc.obj build\temp.win-amd64-3.7\Release\pandas/_libs/src/ujson/lib/ultrajsondec.obj build\temp.win-amd64-3.7\Release\pandas/_libs/tslibs/src/datetime/np_datetime.obj build\temp.win-amd64-3.7\Release\pandas/_libs/tslibs/src/datetime/np_datetime_strings.obj /OUT:c:\Code\pandas_dev\pandas\pandas\_libs\json.cp37-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.7\Release\pandas/_libs/src/ujson/python\json.cp37-win_amd64.lib
   Creating library build\temp.win-amd64-3.7\Release\pandas/_libs/src/ujson/python\json.cp37-win_amd64.lib and object build\temp.win-amd64-3.7\Release\pandas/_libs/src/ujson/python\json.cp37-win_amd64.exp
Generating code
c:\Code\pandas_dev\pandas\pandas\_libs\src\ujson\python\objToJSON.c(181) : warning C4715: 'initObjToJSON': not all control paths return a value
Finished generating code
Finished generating code
Finished generating code
Finished generating code
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.23.28105\\bin\\HostX86\\x64\\cl.exe' failed with exit status 1
```
So in the "link" part, the link is failing because it is still generating code for other modules.  The "cl" part is the compilation that is being done of each module.


Do you actually get an error message anywhere besides the ultimate exit status? Are you seeing that with -j4?
Also note that this only affects the Cythonize piece (i.e. generating C code from the pyx modules) so I'm not sure that would be related to linker errors
With `-j 4`, I don't get the error.  So I think that the `-j 8` doesn't work when it is trying to build the final extensions, after the cythonize step.

So I did see this in the output:
```
c:\Code\pandas_dev\pandas\pandas\_libs\tslibs\src\datetime\np_datetime.c : fatal error C1083: Cannot open compiler generated file: 'c:\Code\pandas_dev\pandas\build\temp.win-amd64-3.7\Release\pandas\_libs\tslibs\src\datetime\np_datetime.obj': Permission denied
```
I think there is a parallelization conflict in that the same file is being compiled by two different processes.  Investigating this

can we confirm this doesn't break things with -j 8?
@jreback The cythonization part with `-j 8` works in parallel just fine.  It's the compilation/linking of the extensions from C to binary that is broken with `-j 8` which is why I opened #30873
@Dr-Irv just to confirm you get that same issue on master with -j8 right?
> @Dr-Irv just to confirm you get that same issue on master with -j8 right?

Yes.  That's why I opened up #30873
is this a common pattern in setup.py?
What is your question in reference to? The `if __name__ == "__main__"` guard or enabling multiprocessing?
> What is your question in reference to? The `if __name__ == "__main__"` guard or enabling multiprocessing?

both - iow is this the standard for parallel building 

if not why are we deviating ?
It might help to separate the conversation of *building* from the conversation of *cythonizing*. This PR deals only with the latter

Cython uses the `multiprocessing` module to cythonize individual files. `multiprocessing` in turn requires the `freeze_support` call as the very first call in a `if __name__ == "__main__"` guard for systems which use the `spawn` method for starting a new process. Traditionally this was just Windows, but as of Python 3.8 it covers macOS as well
thanks @WillAyd