In the manga, it showed he lost it by throwing the little blade of Mihawk, like he did for the cursed sword, but he wasn't ready for even the tiny little knife and he loses his eye.