On a PC, you should be able to sheath your weapon by pressing F. On Xbox, press X. On a PS3, press the square button.