jablonkagroup/ChemBench-Results-dev huggingface dataset:
ChemBench leaderboard entry
Model: Mistral-Large-2
Description: Mistral Large 2 has a 128k context window and supports dozens of languages and along with 80+ coding languages. Mistral Large 2 is designed for single-node inference with long-context applications in mind.
Published: 2024-07-24
Number of parameters: 123000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: no
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.5699426111908178
Analytical Chemistry: 0.48026315789473684
Chemical Preference: 0.5464535464535465
General Chemistry: 0.785234899328859
Inorganic Chemistry: 0.7934782608695652
Materials Science: 0.6666666666666666
Organic Chemistry: 0.7325581395348837
Physical Chemistry: 0.6909090909090909
Technical Chemistry: 0.675
Toxicity and Safety: 0.39555555555555555
ChemBench leaderboard entry
Model: Llama-3.1-70B-Instruct
Description: The Meta Llama 3.1 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction tuned generative models in 8B, 70B and 405B sizes (text in/text out). The Llama 3.1 instruction tuned text only models (8B, 70B, 405B) are optimized for multilingual dialogue use cases and outperform many of the available open source and closed chat models on common industry benchmarks.
Published: 2024-07-23
Number of parameters: 70000000000
Training tokens: 15000000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: no
Alignment method: DPO
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.533715925394548
Analytical Chemistry: 0.40789473684210525
Chemical Preference: 0.5194805194805194
General Chemistry: 0.6912751677852349
Inorganic Chemistry: 0.7717391304347826
Materials Science: 0.6666666666666666
Organic Chemistry: 0.6627906976744186
Physical Chemistry: 0.6424242424242425
Technical Chemistry: 0.65
Toxicity and Safety: 0.3837037037037037
ChemBench leaderboard entry
Model: Claude-3.5 (Sonnet)
Description: Claude models are general purpose large language models. They use a transformer architecture and are trained via unsupervised learning, RLHF, and Constitutional AI (including both a supervised and Reinforcement Learning (RL) phase). Claude 3.5 was developed by Anthropic.
Published: 2024-06-20
Open weights: no
Open dataset: no
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.6255380200860832
Analytical Chemistry: 0.5657894736842105
Chemical Preference: 0.5844155844155844
General Chemistry: 0.825503355704698
Inorganic Chemistry: 0.8369565217391305
Materials Science: 0.7142857142857143
Organic Chemistry: 0.8255813953488372
Physical Chemistry: 0.7696969696969697
Technical Chemistry: 0.85
Toxicity and Safety: 0.44
ChemBench leaderboard entry
Model: Mixtral-8x7b-Instruct (Temperature 1.0)
Description: Mixtral is a sparse mixture-of-experts network. It is a decoder-only model where the feedforward block picks from a set of 8 distinct groups of parameters. At every layer, for every token, a router network chooses two of these groups (the “experts”) to process the token and combine their output additively.
Published: 2023-12-11
Number of parameters: 47000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: yes
Alignment method: DPO
RLHF: no
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.4182209469153515
Analytical Chemistry: 0.27631578947368424
Chemical Preference: 0.5224775224775224
General Chemistry: 0.44966442953020136
Inorganic Chemistry: 0.5108695652173914
Materials Science: 0.40476190476190477
Organic Chemistry: 0.4720930232558139
Physical Chemistry: 0.34545454545454546
Technical Chemistry: 0.325
Toxicity and Safety: 0.26666666666666666
ChemBench leaderboard entry
Model: Command-R+
Description: Cohere Command R is a family of highly scalable language models that balance high performance with strong accuracy. Command-R models were released by Cohere.
Published: 2024-04-04
Number of parameters: 104000000000
Open weights: no
Open dataset: no
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.44763271162123386
Analytical Chemistry: 0.34210526315789475
Chemical Preference: 0.5134865134865135
General Chemistry: 0.4966442953020134
Inorganic Chemistry: 0.5217391304347826
Materials Science: 0.4642857142857143
Organic Chemistry: 0.5511627906976744
Physical Chemistry: 0.32727272727272727
Technical Chemistry: 0.5
Toxicity and Safety: 0.3111111111111111
ChemBench leaderboard entry
Model: GPT-4o React
Description: GPT-4o is OpenAI's third major iteration of their popular large multimodal model, GPT-4, which expands on the capabilities of GPT-4 with Vision.
Published: 2024-05-13
Architecture: DecoderOnly
Open weights: no
Open dataset: no
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: yes
Reasoning model: no
Scores:
Overall: 0.5053802008608321
Analytical Chemistry: 0.46710526315789475
Chemical Preference: 0.4205794205794206
General Chemistry: 0.7583892617449665
Inorganic Chemistry: 0.7282608695652174
Materials Science: 0.5595238095238095
Organic Chemistry: 0.7186046511627907
Physical Chemistry: 0.6
Technical Chemistry: 0.725
Toxicity and Safety: 0.3748148148148148
ChemBench leaderboard entry
Model: Llama-3.1-405B-Instruct
Description: The Meta Llama 3.1 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction tuned generative models in 8B, 70B and 405B sizes (text in/text out). The Llama 3.1 instruction tuned text only models (8B, 70B, 405B) are optimized for multilingual dialogue use cases and outperform many of the available open source and closed chat models on common industry benchmarks.
Published: 2024-07-23
Number of parameters: 405000000000
Training tokens: 15000000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: no
Alignment method: DPO
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.5792682926829268
Analytical Chemistry: 0.506578947368421
Chemical Preference: 0.5404595404595405
General Chemistry: 0.7919463087248322
Inorganic Chemistry: 0.7717391304347826
Materials Science: 0.6547619047619048
Organic Chemistry: 0.7558139534883721
Physical Chemistry: 0.7090909090909091
Technical Chemistry: 0.7
Toxicity and Safety: 0.4192592592592593
ChemBench leaderboard entry
Model: Mixtral-8x7b-Instruct
Description: Mixtral is a sparse mixture-of-experts network. It is a decoder-only model where the feedforward block picks from a set of 8 distinct groups of parameters. At every layer, for every token, a router network chooses two of these groups (the “experts”) to process the token and combine their output additively.
Published: 2023-12-11
Number of parameters: 47000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: yes
Alignment method: DPO
RLHF: no
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.42395982783357244
Analytical Chemistry: 0.26973684210526316
Chemical Preference: 0.5354645354645354
General Chemistry: 0.4228187919463087
Inorganic Chemistry: 0.5543478260869565
Materials Science: 0.4166666666666667
Organic Chemistry: 0.4790697674418605
Physical Chemistry: 0.3333333333333333
Technical Chemistry: 0.325
Toxicity and Safety: 0.2696296296296296
ChemBench leaderboard entry
Model: Llama-3.1-8B-Instruct (Temperature 1.0)
Description: The Meta Llama 3.1 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction tuned generative models in 8B, 70B and 405B sizes (text in/text out). The Llama 3.1 instruction tuned text only models (8B, 70B, 405B) are optimized for multilingual dialogue use cases and outperform many of the available open source and closed chat models on common industry benchmarks.
Published: 2024-07-23
Number of parameters: 8000000000
Training tokens: 15000000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: no
Alignment method: DPO
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.4612625538020086
Analytical Chemistry: 0.3618421052631579
Chemical Preference: 0.5234765234765235
General Chemistry: 0.5302013422818792
Inorganic Chemistry: 0.4782608695652174
Materials Science: 0.4166666666666667
Organic Chemistry: 0.5767441860465117
Physical Chemistry: 0.41818181818181815
Technical Chemistry: 0.4
Toxicity and Safety: 0.32
ChemBench leaderboard entry
Model: GPT-4o
Description: GPT-4o is OpenAI's third major iteration of their popular large multimodal model, GPT-4, which expands on the capabilities of GPT-4 with Vision.
Published: 2024-05-13
Architecture: DecoderOnly
Open weights: no
Open dataset: no
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.610832137733142
Analytical Chemistry: 0.5592105263157895
Chemical Preference: 0.5894105894105894
General Chemistry: 0.8053691275167785
Inorganic Chemistry: 0.8043478260869565
Materials Science: 0.75
Organic Chemistry: 0.7558139534883721
Physical Chemistry: 0.7151515151515152
Technical Chemistry: 0.75
Toxicity and Safety: 0.4414814814814815
ChemBench leaderboard entry
Model: Llama-3-70B-Instruct (Temperature 1.0)
Description: Llama 3 models were trained on a text corpus of over 15T tokens. These models use a tokenizer with a vocabulary of 128K tokens. Additionally, improvements in the post-training procedures substantially reduced false refusal rates, improved alignment, and increased diversity in model responses.
Published: 2024-04-18
Number of parameters: 70000000000
Training tokens: 15000000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: no
Alignment method: DPO
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.5164992826398852
Analytical Chemistry: 0.375
Chemical Preference: 0.5304695304695305
General Chemistry: 0.6040268456375839
Inorganic Chemistry: 0.6847826086956522
Materials Science: 0.6190476190476191
Organic Chemistry: 0.6325581395348837
Physical Chemistry: 0.6
Technical Chemistry: 0.6
Toxicity and Safety: 0.37333333333333335
ChemBench leaderboard entry
Model: PaperQA2
Description: PaperQA2 is a package for doing high-accuracy retrieval augmented generation (RAG) on PDFs or text files, with a focus on the scientific literature. We used PaperQA2 via the non-public API deployed by FutureHouse and the default settings (using Claude-3.5-Sonnet as summarizing and answer-generating LLM).
Published: 2024-09-11
Tool use: yes
Reasoning model: no
Scores:
Overall: 0.5688665710186513
Analytical Chemistry: 0.4605263157894737
Chemical Preference: 0.5634365634365635
General Chemistry: 0.7248322147651006
Inorganic Chemistry: 0.7391304347826086
Materials Science: 0.6904761904761905
Organic Chemistry: 0.6790697674418604
Physical Chemistry: 0.6787878787878788
Technical Chemistry: 0.7
Toxicity and Safety: 0.4237037037037037
ChemBench leaderboard entry
Model: Gemma-1.1-7B-it
Description: Gemma is a family of lightweight, open models built from the research and technology that Google used to create the Gemini models. Gemma models are well-suited for a variety of text generation tasks, including question answering, summarization, and reasoning.
Published: 2024-02-21
Number of parameters: 7000000000
Training tokens: 6000000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: no
Alignment method: PPO
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.19225251076040173
Analytical Chemistry: 0.21052631578947367
Chemical Preference: 0.004995004995004995
General Chemistry: 0.33557046979865773
Inorganic Chemistry: 0.41304347826086957
Materials Science: 0.35714285714285715
Organic Chemistry: 0.37906976744186044
Physical Chemistry: 0.2909090909090909
Technical Chemistry: 0.375
Toxicity and Safety: 0.22962962962962963
ChemBench leaderboard entry
Model: Llama-2-13B Chat
Description: LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. Llama was released by Meta AI.
Published: 2023-07-18
Number of parameters: 13000000000
Training tokens: 2000000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: no
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.25538020086083213
Analytical Chemistry: 0.09210526315789473
Chemical Preference: 0.48451548451548454
General Chemistry: 0.11409395973154363
Inorganic Chemistry: 0.2717391304347826
Materials Science: 0.09523809523809523
Organic Chemistry: 0.15348837209302327
Physical Chemistry: 0.15151515151515152
Technical Chemistry: 0.1
Toxicity and Safety: 0.10074074074074074
ChemBench leaderboard entry
Model: Llama-3.1-8B-Instruct
Description: The Meta Llama 3.1 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction tuned generative models in 8B, 70B and 405B sizes (text in/text out). The Llama 3.1 instruction tuned text only models (8B, 70B, 405B) are optimized for multilingual dialogue use cases and outperform many of the available open source and closed chat models on common industry benchmarks.
Published: 2024-07-23
Number of parameters: 8000000000
Training tokens: 15000000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: no
Alignment method: DPO
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.4716642754662841
Analytical Chemistry: 0.39473684210526316
Chemical Preference: 0.5274725274725275
General Chemistry: 0.5033557046979866
Inorganic Chemistry: 0.5
Materials Science: 0.40476190476190477
Organic Chemistry: 0.5813953488372093
Physical Chemistry: 0.509090909090909
Technical Chemistry: 0.45
Toxicity and Safety: 0.32592592592592595
ChemBench leaderboard entry
Model: Gemma-2-9B-it (Temperature 1.0)
Description: Gemma is a family of lightweight, open models built from the research and technology that Google used to create the Gemini models. Gemma models are well-suited for a variety of text generation tasks, including question answering, summarization, and reasoning.
Published: 2024-06-27
Number of parameters: 9000000000
Training tokens: 8000000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: no
Alignment method: PPO
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.4802725968436155
Analytical Chemistry: 0.2894736842105263
Chemical Preference: 0.5574425574425574
General Chemistry: 0.5570469798657718
Inorganic Chemistry: 0.5434782608695652
Materials Science: 0.5
Organic Chemistry: 0.5465116279069767
Physical Chemistry: 0.4666666666666667
Technical Chemistry: 0.475
Toxicity and Safety: 0.3422222222222222
ChemBench leaderboard entry
Model: Claude-3.5 (Sonnet) React
Description: Claude models are general purpose large language models. They use a transformer architecture and are trained via unsupervised learning, RLHF, and Constitutional AI (including both a supervised and Reinforcement Learning (RL) phase). Claude 3.5 was developed by Anthropic.
Published: 2024-06-20
Open weights: no
Open dataset: no
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: yes
Reasoning model: no
Scores:
Overall: 0.6248206599713056
Analytical Chemistry: 0.5789473684210527
Chemical Preference: 0.5994005994005994
General Chemistry: 0.87248322147651
Inorganic Chemistry: 0.8043478260869565
Materials Science: 0.6785714285714286
Organic Chemistry: 0.8372093023255814
Physical Chemistry: 0.7575757575757576
Technical Chemistry: 0.8
Toxicity and Safety: 0.4088888888888889
ChemBench leaderboard entry
Model: Llama-3.1-70B-Instruct (Temperature 1.0)
Description: The Meta Llama 3.1 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction tuned generative models in 8B, 70B and 405B sizes (text in/text out). The Llama 3.1 instruction tuned text only models (8B, 70B, 405B) are optimized for multilingual dialogue use cases and outperform many of the available open source and closed chat models on common industry benchmarks.
Published: 2024-07-23
Number of parameters: 70000000000
Training tokens: 15000000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: no
Alignment method: DPO
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.5111190817790531
Analytical Chemistry: 0.3684210526315789
Chemical Preference: 0.5354645354645354
General Chemistry: 0.6644295302013423
Inorganic Chemistry: 0.6956521739130435
Materials Science: 0.6547619047619048
Organic Chemistry: 0.5534883720930233
Physical Chemistry: 0.5575757575757576
Technical Chemistry: 0.55
Toxicity and Safety: 0.3896296296296296
ChemBench leaderboard entry
Model: Gemma-2-9B-it
Description: Gemma is a family of lightweight, open models built from the research and technology that Google used to create the Gemini models. Gemma models are well-suited for a variety of text generation tasks, including question answering, summarization, and reasoning.
Published: 2024-06-27
Number of parameters: 9000000000
Training tokens: 8000000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: no
Alignment method: PPO
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.48242467718794835
Analytical Chemistry: 0.3157894736842105
Chemical Preference: 0.5514485514485514
General Chemistry: 0.5436241610738255
Inorganic Chemistry: 0.5543478260869565
Materials Science: 0.5238095238095238
Organic Chemistry: 0.5558139534883721
Physical Chemistry: 0.48484848484848486
Technical Chemistry: 0.525
Toxicity and Safety: 0.33925925925925926
ChemBench leaderboard entry
Model: Llama-2-70B Chat
Description: LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. Llama was released by Meta AI.
Published: 2023-07-18
Number of parameters: 70000000000
Training tokens: 2000000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: no
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.2661406025824964
Analytical Chemistry: 0.07236842105263158
Chemical Preference: 0.4875124875124875
General Chemistry: 0.1342281879194631
Inorganic Chemistry: 0.21739130434782608
Materials Science: 0.17857142857142858
Organic Chemistry: 0.14651162790697675
Physical Chemistry: 0.1696969696969697
Technical Chemistry: 0.125
Toxicity and Safety: 0.1362962962962963
ChemBench leaderboard entry
Model: Llama-3-8B-Instruct
Description: Llama 3 models were trained on a text corpus of over 15T tokens. These models use a tokenizer with a vocabulary of 128K tokens. Additionally, improvements in the post-training procedures substantially reduced false refusal rates, improved alignment, and increased diversity in model responses.
Published: 2024-04-18
Number of parameters: 8000000000
Training tokens: 15000000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: no
Alignment method: DPO
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.45552367288378764
Analytical Chemistry: 0.40789473684210525
Chemical Preference: 0.5154845154845155
General Chemistry: 0.4429530201342282
Inorganic Chemistry: 0.4891304347826087
Materials Science: 0.4166666666666667
Organic Chemistry: 0.5627906976744186
Physical Chemistry: 0.3696969696969697
Technical Chemistry: 0.6
Toxicity and Safety: 0.3244444444444444
ChemBench leaderboard entry
Model: Gemma-1.1-7B-it (Temperature 1.0)
Description: Gemma is a family of lightweight, open models built from the research and technology that Google used to create the Gemini models. Gemma models are well-suited for a variety of text generation tasks, including question answering, summarization, and reasoning.
Published: 2024-02-21
Number of parameters: 7000000000
Training tokens: 6000000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: no
Alignment method: PPO
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.1908177905308465
Analytical Chemistry: 0.21052631578947367
Chemical Preference: 0.008991008991008992
General Chemistry: 0.348993288590604
Inorganic Chemistry: 0.41304347826086957
Materials Science: 0.35714285714285715
Organic Chemistry: 0.37209302325581395
Physical Chemistry: 0.30303030303030304
Technical Chemistry: 0.375
Toxicity and Safety: 0.2162962962962963
ChemBench leaderboard entry
Model: Galactica-120b
Description: Galactica is a large language model developed by Facebook. It is a transformer-based model trained on a large corpus of scientific data.
Published: 2022-11-01
Number of parameters: 120000000000
Training tokens: 450000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: no
RLHF: no
Domain-specific pretraining: yes
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.015066573230553609
Analytical Chemistry: 0
Chemical Preference: 0
General Chemistry: 0.046052631578947366
Inorganic Chemistry: 0.05319148936170213
Materials Science: 0
Organic Chemistry: 0.011337868480725623
Physical Chemistry: 0.055865921787709494
Technical Chemistry: 0
Toxicity and Safety: 0.02318840579710145
ChemBench leaderboard entry
Model: Llama-3-8B-Instruct (Temperature 1.0)
Description: Llama 3 models were trained on a text corpus of over 15T tokens. These models use a tokenizer with a vocabulary of 128K tokens. Additionally, improvements in the post-training procedures substantially reduced false refusal rates, improved alignment, and increased diversity in model responses.
Published: 2024-04-18
Number of parameters: 8000000000
Training tokens: 15000000000000
Architecture: DecoderOnly
Open weights: yes
Open dataset: no
Mixture of experts: no
Alignment method: DPO
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.4565997130559541
Analytical Chemistry: 0.40131578947368424
Chemical Preference: 0.5204795204795205
General Chemistry: 0.436241610738255
Inorganic Chemistry: 0.5434782608695652
Materials Science: 0.4523809523809524
Organic Chemistry: 0.5511627906976744
Physical Chemistry: 0.34545454545454546
Technical Chemistry: 0.625
Toxicity and Safety: 0.3244444444444444
ChemBench leaderboard entry
Model: Gemini-Pro
Description: Gemini models are built from the ground up for multimodality: reasoning seamlessly across text, images, audio, video, and code.
Published: 2024-06-07
Architecture: DecoderOnly
Open weights: no
Open dataset: no
RLHF: yes
Domain-specific pretraining: no
Domain-specific finetuning: no
Tool use: no
Reasoning model: no
Scores:
Overall: 0.45265423242467717
Analytical Chemistry: 0.3881578947368421
Chemical Preference: 0.5004995004995005
General Chemistry: 0.48322147651006714
Inorganic Chemistry: 0.4673913043478261
Materials Science: 0.5
Organic Chemistry: 0.5674418604651162
Physical Chemistry: 0.4484848484848485
Technical Chemistry: 0.475
Toxicity and Safety: 0.30814814814814817