Correct way of loading pretrained model led-base-16384

Krishna_Garg: <p>This <a href="https://github.com/allenai/longformer/issues/192" rel="noopener nofollow ugc">issue</a> discusses about the difference between HuggingFace LED and AllenAI LED. What is the correct way of loading AllenAI’s pretrained model <code>led-base-16384</code>?</p>
<p><code>Approach 1 using HuggingFace LED:</code><br>
Using <code>transformers v4.9.1</code></p>
<pre><code class="lang-auto">from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
  
tokenizer = AutoTokenizer.from_pretrained("allenai/led-base-16384")

model = AutoModelForSeq2SeqLM.from_pretrained("allenai/led-base-16384", gradient_checkpointing=True)
</code></pre>
<p><code>Approach 2 using AllenAI LED:</code><br>
Using transformer version suggested in <code>allenai/longformer/requirements.txt</code>:<br>
<code>git+http://github.com/ibeltagy/transformers.git@longformer_encoder_decoder#egg=transformers</code></p>
<pre><code class="lang-auto">from transformers import AutoTokenizer
from longformer.longformer_encoder_decoder import LongformerEncoderDecoderForConditionalGeneration
tokenizer = AutoTokenizer.from_pretrained("allenai/led-base-16384")
model = LongformerEncoderDecoderForConditionalGeneration.from_pretrained("allenai/led-base-16384", gradient_checkpointing=True)
</code></pre>
<p><code>Results:</code><br>
Approach 1 seems to work but I am not sure whether it is correct because we are loading AllenAI’s pretrained LED using HuggingLED which has a different attention window size.</p>
<p>Approach 2 produces the error:</p>
<pre><code class="lang-auto">  File "/Users/krishna/opt/anaconda3/envs/CiteKP/lib/python3.8/site-packages/transformers/configuration_utils.py", line 353, in get_config_dict
    raise EnvironmentError
OSError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "LED_download.py", line 11, in &lt;module&gt;
    tokenizer = AutoTokenizer.from_pretrained("allenai/led-base-16384")
  File "/Users/krishna/opt/anaconda3/envs/CiteKP/lib/python3.8/site-packages/transformers/tokenization_auto.py", line 209, in from_pretrained
    config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/Users/krishna/opt/anaconda3/envs/CiteKP/lib/python3.8/site-packages/transformers/configuration_auto.py", line 272, in from_pretrained
    config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/Users/krishna/opt/anaconda3/envs/CiteKP/lib/python3.8/site-packages/transformers/configuration_utils.py", line 362, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for 'allenai/led-base-16384'. Make sure that:

- 'allenai/led-base-16384' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'allenai/led-base-16384' is the correct path to a directory containing a config.json file
</code></pre>
<p>I think the error is because the latest pretrained model <code>allenai/led-base-16384</code> is not compatible with the transformers version (i.e. <code>v3.1.0</code>) mentioned in their <code>requirements.txt</code>?</p>

ptrblck: <p>You might want to cross-post the question in the <a href="https://discuss.huggingface.co/">HuggingFace forum</a>, as it seems to be specific to this repository.</p>