Repository: pytorch/vision

Issue:
Using torchvision C++ API without installing python
## ❓ Questions and Help
I have followed the document and built the libtorchvsion.dylib but when I ran `otool -L  libtorchvsion.dylib` it showed
```
libtorchvision.dylib:
	@rpath/libtorchvision.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libc10.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libpython3.8.dylib (compatibility version 3.8.0, current version 3.8.0)
	@rpath/libtorch.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libtorch_cpu.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 902.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
```
Is there a way to get rid of `libpython3.8.dylib`, I think it is required for using python custom operators and we will only use C++.




Discussion:
I managed to build the so with the hack:

https://github.com/pytorch/vision/blob/master/torchvision/csrc/cuda/vision_cuda.h#L2 change this line to include torch/torch.h.

Same with CPU. Also disable the unncessary pybind API call.

+ need change cmake to disable all python3 packages.

However I don't feel like this is the right way to do. Is there any side effect if I disabled all python access?
@mthrok has been looking into something like that for torchaudio and torchtext, maybe his findings could apply for torchvision as well? We might need to move away from `cpp_extensions` for building the C++ extensions, and instead use CMake for that.
I am working on gathering requirements and specing out cmake build for torchtext and torchaudio, which includes both Python extension module and C++ library. I thought torchvision has already have cmake build so did not look into it but if it's something beneficial to torchvision, I can include that one too.

So as to build C++ library, the source code cannot use things that are related to Python. This includes PyBind11 and `<torch/extension.h>`, which, in my understanding, is a wrapper of PyBind11 (I could be wrong here). So it will be easier if we can get rid of `<torch/extension.h>` and replace it with `<torch/script.h>`, but this will involve some code change on Python side too because the way C++ functions are exposed are different. That is, with PyBind11, the extension module is treated as a regular Python module, but TorchScript based binding put all the functions under `torch.ops` or `torch.classes`.

The way I am thinking of cmake build is some thing like the following;
 - `CMakeLists.txt` exposes two targets; C++ library and Python extension.
 - These two targets depend on internal targets which include all the third-party libraries and custom C++ objects that does not include any Python-related code.
 - When building the Python extension, `setup.py` resorts to call `cmake` command with supplement arguments, like Python header/library locations and PyTorch specific configurations.
 - With this `CMakeLists.txt`, one can just add domain library as third party and do `add_subdirectory(vision/text/audio)`.

[Here](https://github.com/pybind/cmake_example/blob/master/setup.py) is an example how Python extension can be build with `cmake` via `setup.py`.
We already solved the `torch.ops` part in torchvision, and we also depend only on `<torch/script.h>`https://github.com/pytorch/vision/blob/15848edbc3c5282ced487a62192fbc763d052019/torchvision/csrc/vision.cpp#L2, but IIRC we had to also include `<Python.h>` for some magical reasons such as https://github.com/pytorch/vision/blob/15848edbc3c5282ced487a62192fbc763d052019/torchvision/csrc/vision.cpp#L19-L35 (although maybe this is legacy and could be removed).
Due to those two points, our CMake file currently depends on Python https://github.com/pytorch/vision/blob/15848edbc3c5282ced487a62192fbc763d052019/CMakeLists.txt#L33 but it shouldn't be the case.
Yeah, `torchtext` also depends on Python because they bind functions via PyBind alongside with TorchScript for performance reason.

I think if they can separate registration (binding) and implementation, then things could work nicely. 
Here is the diagram to illustrate the idea

<img width="943" alt="Screen Shot 2020-09-24 at 13 06 54" src="https://user-images.githubusercontent.com/855818/94177252-40c59a80-fe67-11ea-9667-214bfc4587cd.png">


from [the doc I am working](https://docs.google.com/document/d/1k87YMn43XJz7YwabMVY4s8OTufWEj5mgjijOna6GNtk/edit#) (internal only)
> , but IIRC we had to also include `<Python.h>` for some magical reasons such as
> https://github.com/pytorch/vision/blob/15848edbc3c5282ced487a62192fbc763d052019/torchvision/csrc/vision.cpp#L19-L35
> 
> (although maybe this is legacy and could be removed).

At a glance it looks like you can remove Python2 portion at least.
@mthrok do you have any plan to fix this? Or can I presume it is okay to temporaily applied the fix I mentioned and will have no side effects? 
Could it be a CMake option like USE_PYTHON? When it is disabled, we skip including `<torch/extension.h>` with macro.
> @mthrok do you have any plan to fix this? Or can I presume it is okay to temporaily applied the fix I mentioned and will have no side effects?

I do not have a plan to fix this yet. My work is still proposal phase, and torchvision is more complicated than torchtext or torchaudio.

However if @fmassa's comment 

> We already solved the torch.ops part in torchvision, and we also depend only on <torch/script.h>

is correct, then I think changing `<torch/extension.h>` to `<torch/torch.h>` or `<torch/script.h>` should suffice, and there should be no side effect. (I could be wrong though)

Searching `"pybind11"` reveals [there is one test code](https://github.com/pytorch/vision/search?q=pybind11) that relies on PyBind11, and that code should include `<torch/extension.h>` by itself.
> Could it be a CMake option like USE_PYTHON? When it is disabled, we skip including `<torch/extension.h>` with macro.

That's what I am working on, but in the case of torchvision, `<torch/extension.h>` should not be needed, I presume.
> Searching "pybind11" reveals there is one test code that relies on PyBind11, and that code should include <torch/extension.h> by itself.

Good catch. This is actually an artifact of testing  that should be changed -- we wrapped the C++ models in Python so that we could compare the outputs of the model. But torchvision library itself only uses `torch/script.h`.
@fmassa @mthrok is there any update? Thanks
@stu1130 This is still currently in progress. There are a few PRs that try to minimize the use of `torch/script.h` see https://github.com/pytorch/vision/pull/3087#discussion_r539307310 and #3146.
Hi!

Thanks very much for this project! We built a port of torchvision for the R language: https://github.com/mlverse/torchvision and now we want to work on supporting the C++ operations. For that, it would be great if we didn't require users to install Python in order to use our package.

Would you consider a PR avoiding the Python requirement in the builds?

I can successfully build without Python by simply removing the `#include <Python.h>` statements and adapting the CMake file in order to not link to Python. (see https://github.com/dfalbel/vision/commit/2a81375ab31b8eb0ac5f0175e1720f0daff6d64c)

So my approach would be to have a CMake variable, like `USE_PYTHON` suggested in https://github.com/pytorch/vision/issues/2692#issuecomment-699154008 that would conditionally include Python in the `target_link_libraries` in the CMake file as well as conditionally include `Python.h` in eg. `vision.cpp`

https://github.com/pytorch/vision/blob/68f511eb3510ab6fffd258c8aaeffdc1a574212c/torchvision/csrc/vision.cpp#L3-L5

And other calls like:

https://github.com/pytorch/vision/blob/68f511eb3510ab6fffd258c8aaeffdc1a574212c/torchvision/csrc/io/image/image.cpp#L1-L5

Does that look reasonable? I am no CMake expert, but that seems to be feasible without introducing any breaking change and leaving the default behavior as is.
Hi @dfalbel ,

Thanks for the proposal!

We can accept such a proposal in the CMakeLists (we can actually by default set `USE_PYTHON` to be false in CMakeLists). But note that ideally we wouldn't change the `MOBILE` macro name (otherwise we would need to change a few things elsewhere as well, including our internal mobile builds)

This is something that we would like to get fixed for a while now, see https://github.com/pytorch/vision/issues/3965

But ideally, we would just remove `#include <Python.h>` entirely from our C++ files. It is still there because of some Windows-specific issues (which I believe are related to the fact that we use setuptools to build torchvision extensions).

In any case, we should remove the `#include <Python.h>` by default from the C++ builds.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -201,7 +201,7 @@ def get_extensions():
 
     if sys.platform == "win32":
         define_macros += [("torchvision_EXPORTS", None)]
-
+        define_macros += [("USE_PYTHON", None)]
         extra_compile_args["cxx"].append("/MP")
 
     debug_mode = os.getenv("DEBUG", "0") == "1"
@@ -254,6 +254,9 @@ def get_extensions():
     image_library = []
     image_link_flags = []
 
+    if sys.platform == "win32":
+        image_macros += [("USE_PYTHON", None)]
+
     # Locating libPNG
     libpng = distutils.spawn.find_executable("libpng-config")
     pngfix = distutils.spawn.find_executable("pngfix")


Test patch: