Could you please share the detailed parameters setting for the online demo?
We have tested both the 72B pytorch and AWQ version and the precision is not as good as the online demo. We use xinference with vllm 0.7.3 and transformers 4.49.0 as the backend. We use an example image below
and the prompt below
"
输出每一笔交易的时间和金额，以json形式输出。金额要求准确，无需分隔符，以下是一个示例
##
[{"Time":YYYY-MM-DD, "Amount":0.00},{"Time":YYYY-MM-DD, "Amount":1.00}]
"
is used for both online demo and local deployment. The results are atttached below
for online demo
[
{"Time": "2019-12-08", "Amount": -113.00},
{"Time": "2019-12-08", "Amount": -183.00},
{"Time": "2019-12-18", "Amount": -7500.00},
{"Time": "2019-12-08", "Amount": -301.00},
{"Time": "2019-12-08", "Amount": -63.00},
{"Time": "2019-12-08", "Amount": -776.00},
{"Time": "2019-12-08", "Amount": -1000.00},
{"Time": "2019-12-11", "Amount": -10000.00},
{"Time": "2019-12-11", "Amount": -1486.00},
{"Time": "2019-12-11", "Amount": -596.00},
{"Time": "2019-12-12", "Amount": -1423.00},
{"Time": "2019-12-16", "Amount": -825.00},
{"Time": "2019-12-17", "Amount": -100000.00},
{"Time": "2019-12-18", "Amount": -1322.00},
{"Time": "2019-12-18", "Amount": -4400.00},
{"Time": "2019-12-21", "Amount": 1727.64},
{"Time": "2019-12-24", "Amount": -968.00}
]
and for local deployment
json
[
{"Time":"2019-12-08", "Amount":-113.00},
{"Time":"2019-12-08", "Amount":193.00},
{"Time":"2019-12-10", "Amount":-7500.00},
{"Time":"2019-12-09", "Amount":-301.00},
{"Time":"2019-12-09", "Amount":-63.00},
{"Time":"2019-12-09", "Amount":-754.00},
{"Time":"2019-12-09", "Amount":-1000.00},
{"Time":"2019-12-11", "Amount":-10000.00},
{"Time":"2019-12-11", "Amount":-1496.00},
{"Time":"2019-12-11", "Amount":-906.00},
{"Time":"2019-12-12", "Amount":-1623.00},
{"Time":"2019-12-12", "Amount":-823.00},
{"Time":"2019-12-17", "Amount":-100000.00},
{"Time":"2019-12-19", "Amount":-1322.00},
{"Time":"2019-12-19", "Amount":-4400.00},
{"Time":"2019-12-21", "Amount":1727.64},
{"Time":"2019-12-24", "Amount":-908.00}
]
It can be seen that the online demo has all the amount correct while the local deployment make mistakes e.g. line 2, 9, 10, 11, etc. It seems this is due to a different setting of some parameters such as resolution. Therefore, could you please share the setting for the online demo so that we can re-produce the results in our local deployment