Repository: zulip/zulip

Issue:
Upgrade recommonmark pip package to latest version
update-locked-requirements fails due to compatibility issues. This is probably a dead end since we can't upgrade CommonMark.

```
irtualenv-clone==0.4.0
wcwidth==0.1.7            # via prompt-toolkit
yamole==2.1.6
Could not find a tag or branch '60ed2431c07686a12f2770b2d852c5650f3ccfc6', assuming commit.
Requested libthumbor==1.3.2zulip from git+https://github.com/zulip/libthumbor.git@60ed2431c07686a12f2770b2d852c5650f3ccfc6#egg=libthumbor==1.3.2zulip (from -r requirements/common.in (line 20
8)), but installing version 1.3.2
Could not find a tag or branch '7d8bdc4dbcfcc5a73298747293b99fe53da55315', assuming commit.
Requested talon==1.2.10.zulip1 from git+https://github.com/zulip/talon.git@7d8bdc4dbcfcc5a73298747293b99fe53da55315#egg=talon==1.2.10.zulip1 (from -r requirements/common.in (line 76)), but i
nstalling version 1.2.11
Requested zulip_bots==0.5.8+git from git+https://github.com/zulip/python-zulip-api.git@0.5.8#egg=zulip_bots==0.5.8+git&subdirectory=zulip_bots (from -r requirements/common.in (line 179)), bu
t installing version 0.5.8
Could not find a tag or branch '70ac02bec', assuming commit.
Requested ujson==1.35+git from git+https://github.com/zulip/ultrajson@70ac02bec#egg=ujson==1.35+git (from -r requirements/common.in (line 146)), but installing version 1.35
Requested zulip==0.5.8_git from git+https://github.com/zulip/python-zulip-api.git@0.5.8#egg=zulip==0.5.8_git&subdirectory=zulip (from -r requirements/common.in (line 178)), but installing ve
rsion 0.5.8
Could not find a tag or branch '0d2b15cdb5af5ddec88d41cac19c0f2ce1b1ad38', assuming commit.
Requested django-bitfield==1.9.3+dev.0d2b15cdb5af5ddec88d41cac19c0f2ce1b1ad38 from git+https://github.com/zulip/django-bitfield@0d2b15cdb5af5ddec88d41cac19c0f2ce1b1ad38#egg=django-bitfield==
1.9.3+dev.0d2b15cdb5af5ddec88d41cac19c0f2ce1b1ad38 (from -r requirements/common.in (line 58)), but installing version 1.9.3
Could not find a version that matches CommonMark==0.5.4,>=0.7.3
Tried: 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.5.5, 0.5.5, 0.6.0, 0.6.0, 0.6.1, 0.6.1, 0.6.2, 0.6.2, 0.6.3, 0.6.3, 0.6.4, 0.6.4, 0.7.0, 0.7.0, 0.7.1, 0.7.1, 0.7.2, 0.7.2, 0.7.3, 0.7.3, 0.7.4, 0
.7.4, 0.7.5, 0.7.5, 0.8.0, 0.8.0, 0.8.1, 0.8.1
```
@zulipbot label "area: dependencies"


Discussion:
Hello @zulip/server-dependencies members, this issue was labeled with the "area: dependencies" label, so you may want to check it out!

<!-- areaLabelAddition -->
I tried upgrading this since the dependecy issue with CommonMark got fixed. I am getting the following error.
https://circleci.com/gh/hackerkid/zulip/2978?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link
```
Mar 06 16:09:33 The HTML pages are in _build/html.
Testing only internal links in documentation...
2019-03-06 16:09:39 [scrapy.core.scraper] ERROR: Spider error processing <HEAD https://github.com/zulip/python-zulip-api/blob/master/zulip_bots/README> (referer: None)
Traceback (most recent call last):
  File "/srv/zulip-venv-cache/fd223bc2863de13953a50ef5ef5d691eaca4ebe8/zulip-py3-venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/circleci/zulip/tools/documentation_crawler/documentation_crawler/spiders/common/spiders.py", line 116, in error_callback
    raise Exception('Page not found: {}'.format(response))
Exception: Page not found: <404 https://github.com/zulip/python-zulip-api/blob/master/zulip_bots/README>
2019-03-06 16:09:40 [scrapy.core.scraper] ERROR: Spider error processing <HEAD https://github.com/zulip/zulip/blob/master/requirements/README> (referer: None)
Traceback (most recent call last):
  File "/srv/zulip-venv-cache/fd223bc2863de13953a50ef5ef5d691eaca4ebe8/zulip-py3-venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/circleci/zulip/tools/documentation_crawler/documentation_crawler/spiders/common/spiders.py", line 116, in error_callback
    raise Exception('Page not found: {}'.format(response))
Exception: Page not found: <404 https://github.com/zulip/zulip/blob/master/requirements/README>
2019-03-06 16:09:50 [scrapy.core.scraper] ERROR: Spider error processing <HEAD https://github.com/zulip/zulipbot/blob/master/.github/CONTRIBUTING> (referer: None)
Traceback (most recent call last):
  File "/srv/zulip-venv-cache/fd223bc2863de13953a50ef5ef5d691eaca4ebe8/zulip-py3-venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/circleci/zulip/tools/documentation_crawler/documentation_crawler/spiders/common/spiders.py", line 116, in error_callback
    raise Exception('Page not found: {}'.format(response))
Exception: Page not found: <404 https://github.com/zulip/zulipbot/blob/master/.github/CONTRIBUTING>
Failed!
Exited with code 1
```
It looks like the updated version of RecommonMark is rewriting GitHub link URLs to an invalid version.  
@drrosa this may be a reasonable project for you to pick up.  https://github.com/zulip/zulip/pull/11719 has some potentially relevant context (it seems like possibly the ReadTheDocs toolchain of CommonMark/RecommonMark are somehow trying to solve the same problem we were trying to solve in #11719?)

Needs some investigation.
Sounds good! I'll look into it.
Tonight I got a chance to investigate this issue. Currently, `../tools/build-docs` results in lots of warnings e.g.

`/zulip/docs/tutorials/writing-views.md:367: WARNING: None:any reference target not found: ../subsystems/client.html`

but the docs still build correctly and the links with warnings work just fine.

I then changed that markdown link from `client.html` to `client.md` and that made the warning go away. The link also gets converted correctly to `client.html` after building the docs.

So as @timabbott suspected recommonmark==0.5.0 now allows `.md` links between doc pages! The docs also build correctly with the [latest CommonMark](https://pypi.org/project/commonmark/#history)==0.9.0 version.
``../tools/build-docs`` with the [latest recommonmark](https://pypi.org/project/recommonmark/#history)==0.6.0 gives the following error:

```
Running Sphinx v1.8.4
making output directory...
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 135 source files that are out of date
updating environment: 135 added, 0 changed, 0 removed
/Volumes/Storage/goinfre/drosa-ta/zulip-dev/venv-py3-zulip-rtd/lib/python3.6/site-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
  warn("Container node skipped: type={0}".format(mdnode.t))
/Volumes/Storage/goinfre/drosa-ta/zulip-dev/venv-py3-zulip-rtd/lib/python3.6/site-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
  warn("Container node skipped: type={0}".format(mdnode.t))
/Volumes/Storage/goinfre/drosa-ta/zulip-dev/venv-py3-zulip-rtd/lib/python3.6/site-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
  warn("Container node skipped: type={0}".format(mdnode.t))
/Volumes/Storage/goinfre/drosa-ta/zulip-dev/venv-py3-zulip-rtd/lib/python3.6/site-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
  warn("Container node skipped: type={0}".format(mdnode.t))
/Volumes/Storage/goinfre/drosa-ta/zulip-dev/venv-py3-zulip-rtd/lib/python3.6/site-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
  warn("Container node skipped: type={0}".format(mdnode.t))
/Volumes/Storage/goinfre/drosa-ta/zulip-dev/venv-py3-zulip-rtd/lib/python3.6/site-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
  warn("Container node skipped: type={0}".format(mdnode.t))
/Volumes/Storage/goinfre/drosa-ta/zulip-dev/venv-py3-zulip-rtd/lib/python3.6/site-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
  warn("Container node skipped: type={0}".format(mdnode.t))
/Volumes/Storage/goinfre/drosa-ta/zulip-dev/venv-py3-zulip-rtd/lib/python3.6/site-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
  warn("Container node skipped: type={0}".format(mdnode.t))

Sphinx parallel build error:
TypeError: sequence item 1: expected str instance, NoneType found
```
OK; want to do a prep PR that just upgrades to 0.5.0 without breaking anything?  

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -308,8 +308,18 @@
 # You can specify multiple suffix as a list of string:
 source_suffix = ['.rst', '.md']
 
+# Temporary workaround to supress warnings after upgrading to recommonmark==0.5.0
+# Otherwise, sphinx build complains about all the links ending in .html
+# See PR # for more details
+def on_missing_reference(app, env, node, contnode):
+    if node['reftype'] == 'any':
+        return contnode
+    else:
+        return None
+
 def setup(app: Any) -> None:
 
+    app.connect('missing-reference', on_missing_reference)
     app.add_config_value('recommonmark_config', {
         'enable_eval_rst': True,
         # Turn off recommonmark features we aren't using.


Test patch: