Building docs locally fails

davidradl: <p>I have followed instructions for the Mac on <a href="https://github.com/pytorch/pytorch#from-source" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch#from-source</a> . The instructions for the main build all completed with no errors. I then try to build the docs locally and get an error implying that an Onnix related library is not installed. I could pip install pytorch but I want to use my locally built Pytorch -which should include torch.onnix.</p>
<pre><code class="lang-auto">make html
Traceback (most recent call last):
  File "/Users/davidradley/pytorch/docs/source/scripts/onnx/build_onnx_supported_aten_op_csv_table.py", line 8, in &lt;module&gt;
    from torch.onnx import _onnx_supported_ops
ImportError: cannot import name '_onnx_supported_ops' from 'torch.onnx' (/opt/homebrew/anaconda3/lib/python3.10/site-packages/torch/onnx/__init__.py)
make: *** [onnx] Error 1
</code></pre>
<p>Any thoughts ?</p>

marksaroufim: <p>You might need to just install pytorch from source instead, instructions are on the same page. Another way is to just test in CI if you don’t want to bother building pytorch from source since that can take a bit</p>

davidradl: <p>Thanks for you reply <a class="mention" href="/u/marksaroufim">@marksaroufim</a> . I successfully ran <code>python3 setup.py develop</code> in the root folder of my PyTorch fork. I am new at this - I thought that would have built PyTorch from source - how can I check that the source build worked and will be picked up as a library for the Python script that runs in the docs make file?</p>

marksaroufim: <p>I like to run <code>pip list</code> if you see the line next to <code>torch</code> pointing to a path on your filesystem instead of a version then you know you did things right</p>

davidradl: <p>thanks - good tip <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> . I see <code>torch                         1.12.1      /opt/homebrew/anaconda3/lib/python3.10/site-packages</code> I assume I would expect it to point into my local git repo.</p>

marksaroufim: <p>Yup! That path means you’re not actually set up correctly</p>

marksaroufim: <p>The issue is you’re using <code>python3 setup.py develop</code> - just make it <code>python</code> so you’re using the python environment you installed torch from source in</p>

davidradl: <p><a class="mention" href="/u/marksaroufim">@marksaroufim</a> thanks again for your thoughts:</p>
<p>I ran <code>python setup.py develop</code> and see the same in <code>pip list</code><br>
I run <code>whereis python</code> and get<br>
<code>python: /opt/homebrew/anaconda3/bin/python</code><br>
<code>whereis python3</code> gives me<br>
<code>python3: /usr/bin/python3 /opt/homebrew/share/man/man1/python3.1</code></p>
<p>I have more going on in my environment that is probably relevant. I am using brew pyenv to set 3.10 version of Python. Also I am trying to use conda-forge. So previously ran:</p>
<pre><code class="lang-auto">brew install --cask anaconda
/opt/homebrew/anaconda3/bin/conda install mamba -c conda-forge
/opt/homebrew/anaconda3/bin/conda config --set channel_priority strict
</code></pre>
<p>What I do not understand is</p>
<ol>
<li>
<code>pip list</code> shows I have a torch package (not local, but there is one) so why does the doc make file python file not find the torch onnx package?</li>
<li>Why is the local source not being found?</li>
</ol>
<p>I am wondering whether creating a virtual environment for the make might help isolate the Python environment. This would seem best practise or I am always at the mercy of the python on my machine.</p>

davidradl: <p>I see:<br>
cat  /opt/homebrew/anaconda3/lib/python3.10/site-packages/torch.egg-link</p>
<pre><code class="lang-auto">/Users/davidradley/pytorch
.
</code></pre>
<p>Googling around this format says there should be no new lines in this file format. But this does not seem to matter as the egg-link files are not used for resolve the path for imports.</p>
<p>The paths for imports should be resolved in<br>
<code>cat /opt/homebrew/anaconda3/lib/python3.10/site-packages/easy-install.pth</code></p>
<pre><code class="lang-auto">/Users/davidradley/pytorch
./mpmath-1.2.1-py3.10.egg
/Users/davidradley/pytorch/docs/src/pytorch-sphinx-theme
</code></pre>
<p>So the question is why is <code>/Users/davidradley/pytorch</code> not being picked up.</p>
<p>I run</p>
<pre><code class="lang-auto">python
&gt;&gt;&gt; import sys
&gt;&gt;&gt; sys.path
</code></pre>
<p>and get output<br>
<code>['', '/opt/homebrew/anaconda3/lib/python310.zip', '/opt/homebrew/anaconda3/lib/python3.10', '/opt/homebrew/anaconda3/lib/python3.10/lib-dynload', '/opt/homebrew/anaconda3/lib/python3.10/site-packages', '/opt/homebrew/anaconda3/lib/python3.10/site-packages/PyQt5_sip-12.11.0-py3.10-macosx-11.1-arm64.egg', '/opt/homebrew/anaconda3/lib/python3.10/site-packages/aeosa', '/Users/davidradley/pytorch', '/opt/homebrew/anaconda3/lib/python3.10/site-packages/mpmath-1.2.1-py3.10.egg', '/Users/davidradley/pytorch/docs/src/pytorch-sphinx-theme', '/opt/homebrew/anaconda3/lib/python3.10/site-packages/pycurl-7.45.1-py3.10-macosx-11.1-arm64.egg']</code></p>
<p>It looks like my folder is there, but is not resolving content in my local pytorch folder.</p>
<p>I am uninstalling conda and starting again…</p>

davidradl: <p>I installed miniconda3. I installed <code>make</code> through brew as I did not have one. I followed all the instructions and then issue</p>
<p><code>make html</code> in the docs folder and get</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/Users/davidradley/pytorch/docs/source/scripts/build_opsets.py", line 74, in &lt;module&gt;
    main()
  File "/Users/davidradley/pytorch/docs/source/scripts/build_opsets.py", line 57, in main
    aten_ops_list = get_aten()
  File "/Users/davidradley/pytorch/docs/source/scripts/build_opsets.py", line 19, in get_aten
    parsed_yaml = parse_native_yaml(NATIVE_FUNCTION_YAML_PATH, TAGS_YAML_PATH)
  File "/Users/davidradley/miniconda3/lib/python3.10/site-packages/torchgen/gen.py", line 235, in parse_native_yaml
    _GLOBAL_PARSE_NATIVE_YAML_CACHE[path] = parse_native_yaml_struct(
  File "/Users/davidradley/miniconda3/lib/python3.10/site-packages/torchgen/gen.py", line 167, in parse_native_yaml_struct
    error_check_native_functions(rs)
  File "/Users/davidradley/miniconda3/lib/python3.10/site-packages/torchgen/gen.py", line 277, in error_check_native_functions
    assert len(base_func_map[out_of_place_base_name]) &gt; 0, (
AssertionError: resize_as_ is marked with tag: inplace_view. The codegen expects there to be a corresponding out-of-place view op with the name 'resize_as_' and matching schema, but it didn't find one.
</code></pre>
<p>This is strange as as the assert in gen.py is not intended to be driven for  str(f.func.name) of “resize_as_” due to the if condition    <code>and str(f.func.name) != "resize_as_"</code>.</p>
<pre><code class="lang-auto"> if (
            "inplace_view" in f.tags
            and str(f.func.name) != "resize_"
            and str(f.func.name) != "resize_as_"
        ):
            base_name = f.func.name.name
            overload_name = f.func.name.overload_name
            assert base_name.inplace, (
                f"{f.func.name} is marked with tag: inplace_view, but it doesn't follow the naming "
                "convention for inplace ops - the codegen expects the base name to have a trailing underscore. "
            )
            out_of_place_base_name = BaseOperatorName(
                base_name.base, False, base_name.dunder_method
            )
            assert len(base_func_map[out_of_place_base_name]) &gt; 0, (
                f"{f.func.name} is marked with tag: inplace_view. The codegen expects there to be a corresponding "
                f"out-of-place view op with the name '{base_name}' and matching schema, but it didn't find one. "
            )
</code></pre>
<p>The version of gen.py in miniconda3 does not have the extra  <code>and str(f.func.name) != "resize_as_"</code> condition. But the one in my local file system is correct and has the extra if condition. I am incorrectly picking up the back level version  of gen.py from miniconda3. This seems to be the problem - I now seem to be missing the egg-links for torch an torchgen.</p>

davidradl: <p>I am continuing to update this forum entry in case it helps anyone in the future <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>It looks like I uninstalled brew anaconda and installed the miniconda3. But I am having issues running the<br>
setup.py now as errors. It seems something in my environment is referencing <code>/opt/homebrew/anaconda3/</code> which is now deleted.</p>
<p>There was a warning saying  .conda/environments.txt was not writable, I chmodded it o be writeable and  removed the  anaconda3 environment.</p>
<p>I also ran<br>
<code>python3 setup.py develop --cmake</code><br>
this allowed the make command to run<br>
I reran<br>
<code>python3 setup.py develop</code></p>
<p>I see the torch egg-link is present now.</p>
<pre><code class="lang-auto">ls /Users/davidradley/miniconda3/lib/python3.10/site-packages/*.egg-link

/Users/davidradley/miniconda3/lib/python3.10/site-packages/pytorch-sphinx-theme.egg-link

/Users/davidradley/miniconda3/lib/python3.10/site-packages/torch.egg-link
</code></pre>
<p>there is no egg-link for torchgen - I am guessing this is why it is not picking up the correct python file.<br>
there is also no egg-info folder for torchgen<br>
also there is no entry for torchgen in easy-install.pth</p>
<pre><code class="lang-auto">cat ./lib/python3.10/site-packages/easy-install.pth

/Users/davidradley/pytorch
/Users/davidradley/pytorch/docs/src/pytorch-sphinx-theme
</code></pre>
<p>also the Python sys.path has miniconda before my local repository, which seems the wrong way round.</p>
<p>Any ideas? I wonder what a working system looks like on a system that can make the docs locally.</p>

davidradl: <p>I have raised issue <a href="https://github.com/pytorch/pytorch/issues/104377" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/104377</a> as another person in my team has experienced these symptoms; so this appears to be a regression.</p>
<p>I think I have found what was wrong. setup.py that was not picking up the root folder content for torchgen. See issue for details.</p>