Try to use Box2D physics bodies as the ladders instead of own line segments.