Issues with using SciBERT for Summarizer
Author: WeiyangSun
Pull request: no

Hi All, I am not sure if I am doing this right? But i want my summarizer to use SciBERT SciVocab instead of the traditional BERT vocab. Appreciate any help and advice! This is what I am currently using:

''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

from transformers import BertTokenizer, BertModel

model_version = 'scibert_scivocab_uncased'
do_lower_case = True
model = BertModel.from_pretrained(model_version)
tokenizer = BertTokenizer.from_pretrained(model_version, do_lower_case=do_lower_case)

summarizer = pipeline(task="summarization", model = model, tokenizer = tokenizer)
summary = summarizer(readin_df['Text'][0])

I am facing this error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-39-83180e8b1c13> in <module>
      1 summarizer = pipeline(task="summarization", model = model, tokenizer = tokenizer)
      2 
----> 3 summary = summarizer(readin_df['Text'][0])

/opt/conda/lib/python3.7/site-packages/transformers/pipelines.py in __call__(self, return_tensors, return_text, clean_up_tokenization_spaces, *documents, **generate_kwargs)
   1251 
   1252             summaries = self.model.generate(
-> 1253                 inputs["input_ids"], attention_mask=inputs["attention_mask"], **generate_kwargs,
   1254             )
   1255 

/opt/conda/lib/python3.7/site-packages/torch/autograd/grad_mode.py in decorate_no_grad(*args, **kwargs)
     47         def decorate_no_grad(*args, **kwargs):
     48             with self:
---> 49                 return func(*args, **kwargs)
     50         return decorate_no_grad
     51 

/opt/conda/lib/python3.7/site-packages/transformers/modeling_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, num_return_sequences, attention_mask, decoder_start_token_id)
    788         if self.get_output_embeddings() is None:
    789             raise AttributeError(
--> 790                 "You tried to generate sequences with a model that does not have a LM Head."
    791                 "Please use another model class (e.g. `OpenAIGPTLMHeadModel`, `XLNetLMHeadModel`, `GPT2LMHeadModel`, `CTRLLMHeadModel`, `T5WithLMHeadModel`, `TransfoXLLMHeadModel`, `XLMWithLMHeadModel`, `BartForConditionalGeneration` )"
    792             )

AttributeError: You tried to generate sequences with a model that does not have a LM Head.Please use another model class (e.g. `OpenAIGPTLMHeadModel`, `XLNetLMHeadModel`, `GPT2LMHeadModel`, `CTRLLMHeadModel`, `T5WithLMHeadModel`, `TransfoXLLMHeadModel`, `XLMWithLMHeadModel`, `BartForConditionalGeneration` )

Comments:
I believe only BART and T5 can do summarization for now. See the [documentation regarding the checkpoints that may summarize](https://huggingface.co/transformers/main_classes/pipelines.html#summarizationpipeline).

Here is [a notebook](https://github.com/Nikoschenk/bert-extractive-summarizer/blob/master/colab/scibert-summaries.ipynb) using the scibert model based on the [great code](https://github.com/dmmiller612/bert-extractive-summarizer) that Derek provided.