jablonkagroup/ChemBench-Results-dev huggingface dataset:
ChemBench leaderboard entry
Model: GPT-4
Description: GPT-4 is a large multimodal model released by OpenAI to succeed GPT-3.5 Turbo. It features a context window of 32k tokens.
Published: 2023-03-14
Architecture: DecoderOnly
Open weights: no
Open dataset: no
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.41284074605451937
Analytical Chemistry: 0.4276315789473684
Chemical Preference: 0.16383616383616384
General Chemistry: 0.697986577181208
Inorganic Chemistry: 0.6956521739130435
Materials Science: 0.6071428571428571
Organic Chemistry: 0.6790697674418604
Physical Chemistry: 0.6424242424242425
Technical Chemistry: 0.7
Toxicity and Safety: 0.4103703703703704
ChemBench leaderboard entry
Model: Llama-3-70B-Instruct
Description: Llama 3 models were trained on a text corpus of over 15T tokens. These models use a tokenizer with a vocabulary of 128K tokens. Additionally, improvements in the post-training procedures substantially reduced false refusal rates, improved alignment, and increased diversity in model responses.
Published: 2024-04-18
Number of parameters: 70000000000
Training tokens: 15000000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: no
Alignment method: DPO
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.5179340028694405
Analytical Chemistry: 0.4144736842105263
Chemical Preference: 0.5324675324675324
General Chemistry: 0.6040268456375839
Inorganic Chemistry: 0.6630434782608695
Materials Science: 0.6309523809523809
Organic Chemistry: 0.6325581395348837
Physical Chemistry: 0.593939393939394
Technical Chemistry: 0.625
Toxicity and Safety: 0.3688888888888889
ChemBench leaderboard entry
Model: Phi-3-Medium-4k-Instruct
Description: The Phi-3-Medium-4K-Instruct is a 14B parameters, lightweight, state-of-the-art open model trained with the Phi-3 datasets that includes both synthetic data and the filtered publicly available websites data with a focus on high-quality and reasoning dense properties. The model belongs to the Phi-3 family with the Medium version in two variants 4K and 128K which is the context length (in tokens) that it can support.
Published: 2024-05-21
Number of parameters: 14000000000
Training tokens: 4800000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: no
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.47453371592539456
Analytical Chemistry: 0.34210526315789475
Chemical Preference: 0.5324675324675324
General Chemistry: 0.47651006711409394
Inorganic Chemistry: 0.6304347826086957
Materials Science: 0.5476190476190477
Organic Chemistry: 0.5604651162790698
Physical Chemistry: 0.46060606060606063
Technical Chemistry: 0.55
Toxicity and Safety: 0.33185185185185184
ChemBench leaderboard entry
Model: o1-preview
Description: o1 is trained with reinforcement learning and chain-of-thought reasoning to improve safety, robustness, and reasoning capabilities. The family includes o1-preview and o1-mini versions.
Published: 2024-09-12
Architecture: DecoderOnly
Open weights: no
Open dataset: no
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: yes
Scores:
Overall: 0.6434720229555236
Analytical Chemistry: 0.625
Chemical Preference: 0.5634365634365635
General Chemistry: 0.9328859060402684
Inorganic Chemistry: 0.9021739130434783
Materials Science: 0.7261904761904762
Organic Chemistry: 0.8302325581395349
Physical Chemistry: 0.8484848484848485
Technical Chemistry: 0.85
Toxicity and Safety: 0.47555555555555556
ChemBench leaderboard entry
Model: Claude-3 (Opus)
Description: Claude models are general purpose large language models. They use a transformer architecture and are trained via unsupervised learning, RLHF, and Constitutional AI (including both a supervised and Reinforcement Learning (RL) phase). Claude 3 was developed by Anthropic.
Published: 2024-03-04
Open weights: no
Open dataset: no
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.569583931133429
Analytical Chemistry: 0.46710526315789475
Chemical Preference: 0.5654345654345654
General Chemistry: 0.7651006711409396
Inorganic Chemistry: 0.7934782608695652
Materials Science: 0.6309523809523809
Organic Chemistry: 0.6953488372093023
Physical Chemistry: 0.6484848484848484
Technical Chemistry: 0.7
Toxicity and Safety: 0.4103703703703704
ChemBench leaderboard entry
Model: GPT-3.5 Turbo
Description: GPT-3.5 Turbo, developed by OpenAI, features a context window of 4096 tokens.
Published: 2023-11-06
Architecture: DecoderOnly
Open weights: no
Open dataset: no
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.46628407460545196
Analytical Chemistry: 0.3815789473684211
Chemical Preference: 0.5344655344655345
General Chemistry: 0.4899328859060403
Inorganic Chemistry: 0.5434782608695652
Materials Science: 0.47619047619047616
Organic Chemistry: 0.5883720930232558
Physical Chemistry: 0.4
Technical Chemistry: 0.4
Toxicity and Safety: 0.30962962962962964
ChemBench leaderboard entry
Model: Claude-2
Description: Claude models are general purpose large language models. They use a transformer architecture and are trained via unsupervised learning, RLHF, and Constitutional AI (including both a supervised and Reinforcement Learning (RL) phase). Claude 2 was developed by Anthropic.
Published: 2023-07-11
Open weights: no
Open dataset: no
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.4734576757532281
Analytical Chemistry: 0.375
Chemical Preference: 0.5114885114885115
General Chemistry: 0.5033557046979866
Inorganic Chemistry: 0.6086956521739131
Materials Science: 0.4642857142857143
Organic Chemistry: 0.5930232558139535
Physical Chemistry: 0.503030303030303
Technical Chemistry: 0.475
Toxicity and Safety: 0.33185185185185184