The Night's Watch lost to the wights because they got outnumbered by a large host of White Walkers. As Lord Mormont took only three hundred men to the Fist, they were no match for the wights that flocked to the Fist.