[docs] fixup the MathJax notation in tutorial/layers