Documentation: Update InstallGTM.rst file for errors and new prompts

Fix Syntax errors found during a walkthrough.

Add the prompt for GT.M x64 about removing .o files
and instructions to add libgtmutil.so library to the 'gtmroutines'
env variable instead of the GT.M install directory.

Add non-specific example of o and r entry in gtmroutines setup.

Fix syntax errors.

OSEHRA-Issue: http://issues.osehra.org/browse/OAT-61
Change-Id: I2fc04758f9f8021713a235768d71f2a4702444f9