Repository: zulip/zulip

Issue:
Some of codeblock typeahead doesn't work
There are some codeblock typeaheads (from pygments) that doesn't work when I try it in Zulip, such as `c#`, I think `c#` breaks the codeblock syntax since there is an `#` in it. Btw, we could use `csharp` for alternative of `c#`.

~~--And why there's a `pycon` typeahead, I think it's not a programming language?~~
Edit: `pycon` is a abbreviation of `python console` so it's a normal behaviour

I think we could solve this by removing them (typeahead that doesn't work) from codeblock typeahead list (`tools/setup/lang.json`), and some places that generate those typeaheads..

![peek 2017-06-15 23-41](https://user-images.githubusercontent.com/20320125/27192362-4041332c-5225-11e7-8fd7-dd444d7acc6d.gif)

How to reproduce:

1. Type triple backticks (`) and type c.
2. Then there will appear some options, select c#
3. Type a code, and send the message
Some of codeblock typeahead doesn't work
There are some codeblock typeaheads (from pygments) that doesn't work when I try it in Zulip, such as `c#`, I think `c#` breaks the codeblock syntax since there is an `#` in it. Btw, we could use `csharp` for alternative of `c#`.

~~--And why there's a `pycon` typeahead, I think it's not a programming language?~~
Edit: `pycon` is a abbreviation of `python console` so it's a normal behaviour

I think we could solve this by removing them (typeahead that doesn't work) from codeblock typeahead list (`tools/setup/lang.json`), and some places that generate those typeaheads..

![peek 2017-06-15 23-41](https://user-images.githubusercontent.com/20320125/27192362-4041332c-5225-11e7-8fd7-dd444d7acc6d.gif)

How to reproduce:

1. Type triple backticks (`) and type c.
2. Then there will appear some options, select c#
3. Type a code, and send the message


Discussion:
Hello @zulip/server-compose members, this issue was labeled with the **area: compose** label, so you may want to check it out!
`pycon` seems to be a lexer for the Python Console (read [pygments docs here](http://pygments.org/docs/lexers/?highlight=pycon)), so it's valid.

But I was able to reproduce this error, unfortunately. cc @cory2067 (the implementer of the typeahead syntax)
You can find all the types autocomplete supports at `/static/generated/pygments_data.js`.

After some testing, it seems that that the only non-alphanumeric characters that we support are `+` and `-`.
So things like `c#`, `ambienttalk/2` (whatever that is), `vb.net` have this issue.

We could change the typeahead to ignore the names with special characters, but I feel like that's kind of a cheap way out.
I think this is more so an issue with creating the code blocks themselves.
Well, the pycon typeahead issue is a bit mysterious and we should investigate.  

The main bug here is in the markdown implementation; we're probably a bit too conservative with what other characters we allow to be on the same line.  I suspect we just need to add `/#` to the list in `FENCE_RE` (in the backend), and maybe something similar in the frontend.  

Definitely worth adding tests for both issues when we fix them.
@zulipbot claim
Hello @zulip/server-compose members, this issue was labeled with the **area: compose** label, so you may want to check it out!
`pycon` seems to be a lexer for the Python Console (read [pygments docs here](http://pygments.org/docs/lexers/?highlight=pycon)), so it's valid.

But I was able to reproduce this error, unfortunately. cc @cory2067 (the implementer of the typeahead syntax)
You can find all the types autocomplete supports at `/static/generated/pygments_data.js`.

After some testing, it seems that that the only non-alphanumeric characters that we support are `+` and `-`.
So things like `c#`, `ambienttalk/2` (whatever that is), `vb.net` have this issue.

We could change the typeahead to ignore the names with special characters, but I feel like that's kind of a cheap way out.
I think this is more so an issue with creating the code blocks themselves.
Well, the pycon typeahead issue is a bit mysterious and we should investigate.  

The main bug here is in the markdown implementation; we're probably a bit too conservative with what other characters we allow to be on the same line.  I suspect we just need to add `/#` to the list in `FENCE_RE` (in the backend), and maybe something similar in the frontend.  

Definitely worth adding tests for both issues when we fix them.
@zulipbot claim

Resolving patch:
diff --git a/zerver/lib/bugdown/fenced_code.py b/zerver/lib/bugdown/fenced_code.py
--- a/zerver/lib/bugdown/fenced_code.py
+++ b/zerver/lib/bugdown/fenced_code.py
@@ -98,7 +98,7 @@
     (
         \\{?\\.?
         (?P<lang>
-            [a-zA-Z0-9_+-]*
+            [a-zA-Z0-9_+-./#]*
         ) # "py" or "javascript"
         \\}?
     ) # language, like ".py" or "{javascript}"


Test patch:
diff --git a/frontend_tests/node_tests/markdown.js b/frontend_tests/node_tests/markdown.js
--- a/frontend_tests/node_tests/markdown.js
+++ b/frontend_tests/node_tests/markdown.js
@@ -87,6 +87,15 @@ var social = {
 stream_data.add_sub('Denmark', denmark);
 stream_data.add_sub('social', social);
 
+// Check the default behavior of fenced code blocks
+// works properly before markdown is initialized.
+(function test_fenced_block_defaults() {
+    var input = '\n```\nfenced code\n```\n\nand then after\n';
+    var expected = '\n\n<div class="codehilite"><pre><span></span>fenced code\n</pre></div>\n\n\n\nand then after\n\n';
+    var output = fenced_code.process_fenced_code(input);
+    assert.equal(output, expected);
+}());
+
 var markdown = require('js/markdown.js');
 
 markdown.initialize();
@@ -181,6 +190,10 @@ var bugdown_data = JSON.parse(fs.readFileSync(path.join(__dirname, '../../zerver
          expected: '<div class="codehilite"><pre><span></span>    fenced code trailing whitespace\n</pre></div>\n\n\n<p>and then after</p>'},
         {input: '* a\n* list \n* here',
          expected: '<ul>\n<li>a</li>\n<li>list </li>\n<li>here</li>\n</ul>'},
+        {input: '\n```c#\nfenced code special\n```\n\nand then after\n',
+         expected: '<div class="codehilite"><pre><span></span>fenced code special\n</pre></div>\n\n\n<p>and then after</p>'},
+        {input: '\n```vb.net\nfenced code dot\n```\n\nand then after\n',
+         expected: '<div class="codehilite"><pre><span></span>fenced code dot\n</pre></div>\n\n\n<p>and then after</p>'},
         {input: 'Some text first\n* a\n* list \n* here\n\nand then after',
          expected: '<p>Some text first</p>\n<ul>\n<li>a</li>\n<li>list </li>\n<li>here</li>\n</ul>\n<p>and then after</p>'},
         {input: '1. an\n2. ordered \n3. list',
diff --git a/zerver/tests/test_bugdown.py b/zerver/tests/test_bugdown.py
--- a/zerver/tests/test_bugdown.py
+++ b/zerver/tests/test_bugdown.py
@@ -110,10 +110,13 @@ def test_serial_code(self):
             'hello()',
             '```',
             '',
-            '``` .py',
+            '```vb.net',
             'goodbye()',
             '```',
             '',
+            '```c#',
+            'weirdchar()',
+            '```',
             ''
         ]
         expected = [
@@ -122,8 +125,11 @@ def test_serial_code(self):
             '',
             '',
             '',
-            '**py:goodbye()**',
+            '**vb.net:goodbye()**',
+            '',
+            '',
             '',
+            '**c#:weirdchar()**',
             '',
             ''
         ]