Issue 1:
🚀 Feature request

The current implementation of TokenClassificationPipeline cannot do anything with text input beyond the length (measured in tokens) allowed by the model's number of positional embeddings. I think it would be useful to many users to have a setting in the pipeline that allows for a "sliding window" approach that can take documents longer than the .config.max_position_embeddings of the model in use.

The modified version of TokenClassificationPipeline could be instantiated with the optional new params window_size (defaults to model.max_position_embeddings - 2 to accomodate the [CLS] and [SEP] special tokens) and stride (defaults to window_size / 2, the user can disable the sliding window by setting stride = 0) like so:

model, tokenizer = # Some model, some tokenizer
pipeline = TokenClassificationPipeline(model=model, tokenizer=tokenizer, aggregation_strategy='first', stride=254)

Motivation

I run tasks that involve doing named entity recognition on longer documents. I have to use my own modified subclass of TokenClassificationPipeline in order to do so, since the existing class does not support processing texts longer than the maximum number of position embeddings for the underlying models.

Your contribution

I will submit a PR adding sliding window functionality to TokenClassificationPipeline. I have previously created a sliding window implementation of Transformers' NER Pipeline here, and will re-work this to work with TensorFlow and meet the code standards of HuggingFace/Transformers.

Issue 2:
Hi, I am trying to convert a fine-tuned GPT-Neo (125M) model to ONNX using the code below:

from transformers import pipeline, convert_graph_to_onnx, GPTNeoForCausalLM, GPT2Tokenizer
from pathlib import Path
import torch

model_name = "EleutherAI/gpt-neo-125M"
pipeline_name = "text-generation"

tokenizer = GPT2Tokenizer.from_pretrained("EleutherAI/gpt-neo-125M", bos_token='
',
eos_token='
', pad_token='
')
nlp = pipeline(pipeline_name, model=model_name, tokenizer=tokenizer)

with torch.no_grad():
(
input_names,
output_names,
dynamic_axes,
tokens,
) = convert_graph_to_onnx.infer_shapes(nlp, "pt")
ordered_input_names, model_args = convert_graph_to_onnx.ensure_valid_input(
nlp.model, tokens, input_names
)

model_name = 'gpt-neo'

predictor_path = './' + model_name

model2 = GPTNeoForCausalLM.from_pretrained(predictor_path)

text = "I feel happy and "
prompt = f'
Review: {text} Sentiment: 
'
encodings_dict = nlp.tokenizer(prompt, truncation=True, max_length=300, padding="max_length", return_tensors="pt")

torch.onnx.export(
model2,
(encodings_dict['input_ids'], encodings_dict['attention_mask']),
'model_test.onnx',
input_names=input_names,
output_names=output_names,
dynamic_axes=dynamic_axes,
do_constant_folding=True,
use_external_data_format=True, # Needed because of model size
enable_onnx_checker=True,
opset_version=13
)

But I get this error:

ValueError Traceback (most recent call last)
<ipython-input-15-024e093371a4> in <module>()
43 use_external_data_format=True, # Needed because of model size
44 enable_onnx_checker=True,
---> 45 opset_version=13
46 )

3 frames
/usr/local/lib/python3.7/dist-packages/torch/onnx/utils.py in _validate_dynamic_axes(dynamic_axes, model, input_names, output_names)
1300 for i, x in enumerate(value):
1301 if not isinstance(x, int):
-> 1302 raise ValueError("The type of axis index is expected to be an integer")
1303 if x in value_dict:
1304 warnings.warn("Duplicate dynamic axis index {} was provided for input {}."

ValueError: The type of axis index is expected to be an integer

But if I remove dynamic_axes, I get this:

IndexError Traceback (most recent call last)
<ipython-input-16-89ddee8c10f8> in <module>()
43 use_external_data_format=True, # Needed because of model size
44 enable_onnx_checker=True,
---> 45 opset_version=13
46 )

15 frames
/usr/local/lib/python3.7/dist-packages/transformers/models/gpt_neo/modeling_gpt_neo.py in forward(self, input_ids, past_key_values, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict)
545 past_key_values = tuple([None] * len(self.h))
546 else:
--> 547 past_length = past_key_values[0][0].size(-2)
548
549 device = input_ids.device if input_ids is not None else inputs_embeds.device

IndexError: dimension specified as -2 but tensor has no dimensions

Can someone help me please?