Need help replicating public test set results
Hi again. I've been trying to replicate the leaderboard scores in the public test set (https://vilmedic.app/misc/bionlp24/leaderboard) for both findings and impression, but my scores are a bit below what is shown in the leaderboard. The code I'm using for this is available here: https://colab.research.google.com/drive/17gJYylny8ZgOMr_9FOeOSOXhQHnTVFeL?usp=sharing
Do you know if it's possible to replicate the leaderboard scores using Colab? If yes, how would I need to tweak the code in order to get the same (or very similar) scores?
Thanks in advance.
Pablo
Hi @pamessina ,
I used the following during the competition for the findings and impression sections (no temperature). See how that changes the scores.
Generate both sections:
output_ids = model.generate(
pixel_values=batch['images'].to(device=device),
max_length=512,
num_beams=4,
do_sample=True,
bad_words_ids=[[tokenizer.convert_tokens_to_ids('[NF]')], [tokenizer.convert_tokens_to_ids('[NI]')]],
)
findings, impression = model.split_and_decode_sections(output_ids, tokenizer)
_ = [print(f'Findings:\n\t{i}\nImpression:\n\t{j}\n') for i,j in zip(findings, impression)]
Also, do_sample=False