The weakest faction in terms of how easy they are to siege would probably be the Khergits or Sarranids, as both tend to rely on horses, which have no place in sieges. They also seem to have much lighter armor.