More fixes. Can run inference w/ factors, but behavior is not yet what I expect.