Fix markup in math operator documentation

In the documentation, surround the new operators with `<tt>` tags instead of Markdown-style backticks, since this is HTML.
Merge pull request #3336 from roryokane/fix-math-operator-docs-markup

Fix markup in math operator documentation