allenai/OLMo-1B · Warning about some weights were not initialized

May 2

I am using lm_eval from lm-evaluation-harness to get responses.

lm_eval \
    --model hf \
    --model_args pretrained=allenai/OLMo-1B,revision=step738000-tokens3095B \
    --tasks mmlu \
    --num_fewshot 0 \
    --batch_size auto \
    --output_path mmlu/olmo.jsonl \
    --log_samples \
    --trust_remote_code

Then, it shows the following warning:

Some weights of OLMoForCausalLM were not initialized from the model checkpoint at allenai/OLMo-1B and are newly initialized: ['model.transformer.ff_out.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Is this behavior expected for the above lm_eval parameters? What can be the cause of this warning? Because I am getting a lower accuracy score for MMLU than I expected.

shanearora

Allen Institute for AI org May 2

•

edited May 2

Could you share your installed versions of transformers and ai2_olmo? I suspect that you need to update the latter.

dumitrac

May 2

Hi @yusuf-ackan - I just tried this (on Ubuntu 24.04 with Python 3.12.3):

$ python -m venv .venv
$ source .venv/bin/activate
$ python -m pip install lm_eval
$ python -m pip install ai2_olmo
$ <copy-paste your lm_eval command>

The evaluation succeeds and nothing is logged about "init".
Could you please share more about your setup?

yusuf-ackan

May 2

Hi, previously, ai2_olmo=0.2.5 but after the update to 0.3.0, the issue was resolved. Thank you

yusuf-ackan changed discussion status to closed May 2