Med-RewardBench Inference and Evaluation

Data

Please download the dataset from Nandzy/Med-Reward.

Inference

We recommend using vLLM as the inference engine for optimal performance. Once you have deployed your model using vLLM (ensuring the port is set to 8000), you can use 4_api_benchmark_multi.py to run inference on the dataset.

python 4_api_benchmark_multi.py
--model "Qwen/Qwen2-VL-7B-Instruct"

Evaluation

To assess the model's performance, use 5_calculate_accuracy.py. Update the script with the correct data path corresponding to the output file.

python 5_calculate_accuracy.py --output_file "reaults.txt"

This will compute relevant metrics and provide insights into the model's effectiveness.