Remove tmp files for factor graph construction