Nyströmformer

## [](#overview)Overview

The Nyströmformer model was proposed in [_Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention_](https://arxiv.org/abs/2102.03902) by Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh.

The abstract from the paper is the following:

_Transformers have emerged as a powerful tool for a broad range of natural language processing tasks. A key component that drives the impressive performance of Transformers is the self-attention mechanism that encodes the influence or dependence of other tokens on each specific token. While beneficial, the quadratic complexity of self-attention on the input sequence length has limited its application to longer sequences — a topic being actively studied in the community. To address this limitation, we propose Nyströmformer — a model that exhibits favorable scalability as a function of sequence length. Our idea is based on adapting the Nyström method to approximate standard self-attention with O(n) complexity. The scalability of Nyströmformer enables application to longer sequences with thousands of tokens. We perform evaluations on multiple downstream tasks on the GLUE benchmark and IMDB reviews with standard sequence length, and find that our Nyströmformer performs comparably, or in a few cases, even slightly better, than standard self-attention. On longer sequence tasks in the Long Range Arena (LRA) benchmark, Nyströmformer performs favorably relative to other efficient self-attention methods. Our code is available at this https URL._

This model was contributed by [novice03](https://huggingface.co/novice03). The original code can be found [here](https://github.com/mlpen/Nystromformer).

## [](#documentation-resources)Documentation resources

-   [Text classification task guide](../tasks/sequence_classification)
-   [Token classification task guide](../tasks/token_classification)
-   [Question answering task guide](../tasks/question_answering)
-   [Masked language modeling task guide](../tasks/masked_language_modeling)
-   [Multiple choice task guide](../tasks/multiple_choice)

## [](#transformers.NystromformerConfig)NystromformerConfig

### class transformers.NystromformerConfig

[](#transformers.NystromformerConfig)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/nystromformer/configuration_nystromformer.py#L29)

( vocab\_size = 30000hidden\_size = 768num\_hidden\_layers = 12num\_attention\_heads = 12intermediate\_size = 3072hidden\_act = 'gelu\_new'hidden\_dropout\_prob = 0.1attention\_probs\_dropout\_prob = 0.1max\_position\_embeddings = 510type\_vocab\_size = 2segment\_means\_seq\_len = 64num\_landmarks = 64conv\_kernel\_size = 65inv\_coeff\_init\_option = Falseinitializer\_range = 0.02layer\_norm\_eps = 1e-05pad\_token\_id = 1bos\_token\_id = 0eos\_token\_id = 2\*\*kwargs )

This is the configuration class to store the configuration of a [NystromformerModel](/docs/transformers/v4.30.0/en/model_doc/nystromformer#transformers.NystromformerModel). It is used to instantiate an Nystromformer model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Nystromformer [uw-madison/nystromformer-512](https://huggingface.co/uw-madison/nystromformer-512) architecture.

Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the documentation from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

Example:

```
>>> from transformers import NystromformerModel, NystromformerConfig

>>> 
>>> configuration = NystromformerConfig()

>>> 
>>> model = NystromformerModel(configuration)

>>> 
>>> configuration = model.config```

## [](#transformers.NystromformerModel)NystromformerModel

### class transformers.NystromformerModel

[](#transformers.NystromformerModel)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/nystromformer/modeling_nystromformer.py#L549)

( config )

Parameters

-   [](#transformers.NystromformerModel.config)**config** ([NystromformerConfig](/docs/transformers/v4.30.0/en/model_doc/nystromformer#transformers.NystromformerConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

The bare Nyströmformer Model transformer outputting raw hidden-states without any specific head on top. This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.NystromformerModel.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/nystromformer/modeling_nystromformer.py#L574)

( input\_ids: typing.Optional\[torch.LongTensor\] = Noneattention\_mask: typing.Optional\[torch.FloatTensor\] = Nonetoken\_type\_ids: typing.Optional\[torch.LongTensor\] = Noneposition\_ids: typing.Optional\[torch.LongTensor\] = Nonehead\_mask: typing.Optional\[torch.FloatTensor\] = Noneinputs\_embeds: typing.Optional\[torch.FloatTensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → [transformers.modeling\_outputs.BaseModelOutputWithPastAndCrossAttentions](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPastAndCrossAttentions) or `tuple(torch.FloatTensor)`

The [NystromformerModel](/docs/transformers/v4.30.0/en/model_doc/nystromformer#transformers.NystromformerModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

```
>>> from transformers import AutoTokenizer, NystromformerModel
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("uw-madison/nystromformer-512")
>>> model = NystromformerModel.from_pretrained("uw-madison/nystromformer-512")

>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> outputs = model(**inputs)

>>> last_hidden_states = outputs.last_hidden_state```

## [](#transformers.NystromformerForMaskedLM)NystromformerForMaskedLM

### class transformers.NystromformerForMaskedLM

[](#transformers.NystromformerForMaskedLM)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/nystromformer/modeling_nystromformer.py#L660)

( config )

Parameters

-   [](#transformers.NystromformerForMaskedLM.config)**config** ([NystromformerConfig](/docs/transformers/v4.30.0/en/model_doc/nystromformer#transformers.NystromformerConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

Nyströmformer Model with a `language modeling` head on top. This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.NystromformerForMaskedLM.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/nystromformer/modeling_nystromformer.py#L678)

( input\_ids: typing.Optional\[torch.LongTensor\] = Noneattention\_mask: typing.Optional\[torch.FloatTensor\] = Nonetoken\_type\_ids: typing.Optional\[torch.LongTensor\] = Noneposition\_ids: typing.Optional\[torch.LongTensor\] = Nonehead\_mask: typing.Optional\[torch.FloatTensor\] = Noneinputs\_embeds: typing.Optional\[torch.FloatTensor\] = Nonelabels: typing.Optional\[torch.LongTensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → [transformers.modeling\_outputs.MaskedLMOutput](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.MaskedLMOutput) or `tuple(torch.FloatTensor)`

The [NystromformerForMaskedLM](/docs/transformers/v4.30.0/en/model_doc/nystromformer#transformers.NystromformerForMaskedLM) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

```
>>> from transformers import AutoTokenizer, NystromformerForMaskedLM
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("uw-madison/nystromformer-512")
>>> model = NystromformerForMaskedLM.from_pretrained("uw-madison/nystromformer-512")

>>> inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt")

>>> with torch.no_grad():
...     logits = model(**inputs).logits

>>> 
>>> mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]

>>> predicted_token_id = logits[0, mask_token_index].argmax(axis=-1)

>>> labels = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"]
>>> 
>>> labels = torch.where(inputs.input_ids == tokenizer.mask_token_id, labels, -100)

>>> outputs = model(**inputs, labels=labels)```

## [](#transformers.NystromformerForSequenceClassification)NystromformerForSequenceClassification

### class transformers.NystromformerForSequenceClassification

[](#transformers.NystromformerForSequenceClassification)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/nystromformer/modeling_nystromformer.py#L765)

( config )

Parameters

-   [](#transformers.NystromformerForSequenceClassification.config)**config** ([NystromformerConfig](/docs/transformers/v4.30.0/en/model_doc/nystromformer#transformers.NystromformerConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

Nyströmformer Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks.

This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.NystromformerForSequenceClassification.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/nystromformer/modeling_nystromformer.py#L775)

( input\_ids: typing.Optional\[torch.LongTensor\] = Noneattention\_mask: typing.Optional\[torch.FloatTensor\] = Nonetoken\_type\_ids: typing.Optional\[torch.LongTensor\] = Noneposition\_ids: typing.Optional\[torch.LongTensor\] = Nonehead\_mask: typing.Optional\[torch.FloatTensor\] = Noneinputs\_embeds: typing.Optional\[torch.FloatTensor\] = Nonelabels: typing.Optional\[torch.LongTensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → [transformers.modeling\_outputs.SequenceClassifierOutput](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.SequenceClassifierOutput) or `tuple(torch.FloatTensor)`

The [NystromformerForSequenceClassification](/docs/transformers/v4.30.0/en/model_doc/nystromformer#transformers.NystromformerForSequenceClassification) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example of single-label classification:

```
>>> import torch
>>> from transformers import AutoTokenizer, NystromformerForSequenceClassification

>>> tokenizer = AutoTokenizer.from_pretrained("uw-madison/nystromformer-512")
>>> model = NystromformerForSequenceClassification.from_pretrained("uw-madison/nystromformer-512")

>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")

>>> with torch.no_grad():
...     logits = model(**inputs).logits

>>> predicted_class_id = logits.argmax().item()

>>> 
>>> num_labels = len(model.config.id2label)
>>> model = NystromformerForSequenceClassification.from_pretrained("uw-madison/nystromformer-512", num_labels=num_labels)

>>> labels = torch.tensor([1])
>>> loss = model(**inputs, labels=labels).loss```

Example of multi-label classification:

```
>>> import torch
>>> from transformers import AutoTokenizer, NystromformerForSequenceClassification

>>> tokenizer = AutoTokenizer.from_pretrained("uw-madison/nystromformer-512")
>>> model = NystromformerForSequenceClassification.from_pretrained("uw-madison/nystromformer-512", problem_type="multi_label_classification")

>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")

>>> with torch.no_grad():
...     logits = model(**inputs).logits

>>> predicted_class_ids = torch.arange(0, logits.shape[-1])[torch.sigmoid(logits).squeeze(dim=0) > 0.5]

>>> 
>>> num_labels = len(model.config.id2label)
>>> model = NystromformerForSequenceClassification.from_pretrained(
...     "uw-madison/nystromformer-512", num_labels=num_labels, problem_type="multi_label_classification"
... )

>>> labels = torch.sum(
...     torch.nn.functional.one_hot(predicted_class_ids[None, :].clone(), num_classes=num_labels), dim=1
... ).to(torch.float)
>>> loss = model(**inputs, labels=labels).loss```

## [](#transformers.NystromformerForMultipleChoice)NystromformerForMultipleChoice

### class transformers.NystromformerForMultipleChoice

[](#transformers.NystromformerForMultipleChoice)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/nystromformer/modeling_nystromformer.py#L858)

( config )

Parameters

-   [](#transformers.NystromformerForMultipleChoice.config)**config** ([NystromformerConfig](/docs/transformers/v4.30.0/en/model_doc/nystromformer#transformers.NystromformerConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

Nyströmformer Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks.

This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.NystromformerForMultipleChoice.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/nystromformer/modeling_nystromformer.py#L869)

( input\_ids: typing.Optional\[torch.LongTensor\] = Noneattention\_mask: typing.Optional\[torch.FloatTensor\] = Nonetoken\_type\_ids: typing.Optional\[torch.LongTensor\] = Noneposition\_ids: typing.Optional\[torch.LongTensor\] = Nonehead\_mask: typing.Optional\[torch.FloatTensor\] = Noneinputs\_embeds: typing.Optional\[torch.FloatTensor\] = Nonelabels: typing.Optional\[torch.LongTensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → [transformers.modeling\_outputs.MultipleChoiceModelOutput](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.MultipleChoiceModelOutput) or `tuple(torch.FloatTensor)`

The [NystromformerForMultipleChoice](/docs/transformers/v4.30.0/en/model_doc/nystromformer#transformers.NystromformerForMultipleChoice) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

```
>>> from transformers import AutoTokenizer, NystromformerForMultipleChoice
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("uw-madison/nystromformer-512")
>>> model = NystromformerForMultipleChoice.from_pretrained("uw-madison/nystromformer-512")

>>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
>>> choice0 = "It is eaten with a fork and a knife."
>>> choice1 = "It is eaten while held in the hand."
>>> labels = torch.tensor(0).unsqueeze(0)  

>>> encoding = tokenizer([prompt, prompt], [choice0, choice1], return_tensors="pt", padding=True)
>>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels)  

>>> 
>>> loss = outputs.loss
>>> logits = outputs.logits```

## [](#transformers.NystromformerForTokenClassification)NystromformerForTokenClassification

### class transformers.NystromformerForTokenClassification

[](#transformers.NystromformerForTokenClassification)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/nystromformer/modeling_nystromformer.py#L953)

( config )

Parameters

-   [](#transformers.NystromformerForTokenClassification.config)**config** ([NystromformerConfig](/docs/transformers/v4.30.0/en/model_doc/nystromformer#transformers.NystromformerConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

Nyströmformer Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks.

This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.NystromformerForTokenClassification.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/nystromformer/modeling_nystromformer.py#L965)

( input\_ids: typing.Optional\[torch.LongTensor\] = Noneattention\_mask: typing.Optional\[torch.FloatTensor\] = Nonetoken\_type\_ids: typing.Optional\[torch.LongTensor\] = Noneposition\_ids: typing.Optional\[torch.LongTensor\] = Nonehead\_mask: typing.Optional\[torch.FloatTensor\] = Noneinputs\_embeds: typing.Optional\[torch.FloatTensor\] = Nonelabels: typing.Optional\[torch.LongTensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → [transformers.modeling\_outputs.TokenClassifierOutput](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.TokenClassifierOutput) or `tuple(torch.FloatTensor)`

The [NystromformerForTokenClassification](/docs/transformers/v4.30.0/en/model_doc/nystromformer#transformers.NystromformerForTokenClassification) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

```
>>> from transformers import AutoTokenizer, NystromformerForTokenClassification
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("uw-madison/nystromformer-512")
>>> model = NystromformerForTokenClassification.from_pretrained("uw-madison/nystromformer-512")

>>> inputs = tokenizer(
...     "HuggingFace is a company based in Paris and New York", add_special_tokens=False, return_tensors="pt"
... )

>>> with torch.no_grad():
...     logits = model(**inputs).logits

>>> predicted_token_class_ids = logits.argmax(-1)

>>> 
>>> 
>>> 
>>> predicted_tokens_classes = [model.config.id2label[t.item()] for t in predicted_token_class_ids[0]]

>>> labels = predicted_token_class_ids
>>> loss = model(**inputs, labels=labels).loss```

## [](#transformers.NystromformerForQuestionAnswering)NystromformerForQuestionAnswering

### class transformers.NystromformerForQuestionAnswering

[](#transformers.NystromformerForQuestionAnswering)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/nystromformer/modeling_nystromformer.py#L1031)

( config )

Parameters

-   [](#transformers.NystromformerForQuestionAnswering.config)**config** ([NystromformerConfig](/docs/transformers/v4.30.0/en/model_doc/nystromformer#transformers.NystromformerConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

Nyströmformer Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`).

This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.NystromformerForQuestionAnswering.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/nystromformer/modeling_nystromformer.py#L1044)

( input\_ids: typing.Optional\[torch.LongTensor\] = Noneattention\_mask: typing.Optional\[torch.FloatTensor\] = Nonetoken\_type\_ids: typing.Optional\[torch.LongTensor\] = Noneposition\_ids: typing.Optional\[torch.LongTensor\] = Nonehead\_mask: typing.Optional\[torch.FloatTensor\] = Noneinputs\_embeds: typing.Optional\[torch.FloatTensor\] = Nonestart\_positions: typing.Optional\[torch.LongTensor\] = Noneend\_positions: typing.Optional\[torch.LongTensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → [transformers.modeling\_outputs.QuestionAnsweringModelOutput](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.QuestionAnsweringModelOutput) or `tuple(torch.FloatTensor)`

The [NystromformerForQuestionAnswering](/docs/transformers/v4.30.0/en/model_doc/nystromformer#transformers.NystromformerForQuestionAnswering) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

```
>>> from transformers import AutoTokenizer, NystromformerForQuestionAnswering
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("uw-madison/nystromformer-512")
>>> model = NystromformerForQuestionAnswering.from_pretrained("uw-madison/nystromformer-512")

>>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"

>>> inputs = tokenizer(question, text, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)

>>> answer_start_index = outputs.start_logits.argmax()
>>> answer_end_index = outputs.end_logits.argmax()

>>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]

>>> 
>>> target_start_index = torch.tensor([14])
>>> target_end_index = torch.tensor([15])

>>> outputs = model(**inputs, start_positions=target_start_index, end_positions=target_end_index)
>>> loss = outputs.loss```