make factor graph building to be lazy