README.md

---
base_model: google/pegasus-x-base
tags:
- generated_from_trainer
datasets:
- arxiv-summarization

widget:
- text: >-
    
    [Abstract] The dominant sequence transduction models are based on complex
    recurrent or convolutional neural networks in an encoder-decoder
    configuration. The best performing models also connect the encoder and
    decoder through an attention mechanism. We propose a new simple network
    architecture, the Transformer, based solely on attention mechanisms,
    dispensing with recurrence and convolutions entirely. Experiments on two
    machine translation tasks show these models to be superior in quality while
    being more parallelizable and requiring significantly less time to train.
    Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation
    task, improving over the existing best results, including ensembles by over
    2 BLEU. On the WMT 2014 English-to-French translation task, our model
    establishes a new single-model state-of-the-art BLEU score of 41.8 after
    training for 3.5 days on eight GPUs, a small fraction of the training costs
    of the best models from the literature. We show that the Transformer
    generalizes well to other tasks by applying it successfully to English
    constituency parsing both with large and limited training data.
    [Introduction] Recurrent neural networks, long short-term memory [13] and
    gated recurrent [7] neural networks in particular, have been firmly
    established as state of the art approaches in sequence modeling and
    transduction problems such as language modeling and machine translation [35,
    2, 5]. Numerous efforts have since continued to push the boundaries of
    recurrent language models and encoder-decoder architectures [38, 24, 15].
    Recurrent models typically factor computation along the symbol positions of
    the input and output sequences. Aligning the positions to steps in
    computation time, they generate a sequence of hidden states ht, as a
    function of the previous hidden state ht−1 and the input for position t.
    This inherently sequential nature precludes parallelization within training
    examples, which becomes critical at longer sequence lengths, as memory
    constraints limit batching across examples. Recent work has achieved
    significant improvements in computational efficiency through factorization
    tricks [21] and conditional computation [32], while also improving model
    performance in case of the latter. The fundamental constraint of sequential
    computation, however, remains. Attention mechanisms have become an integral
    part of compelling sequence modeling and transduction models in various
    tasks, allowing modeling of dependencies without regard to their distance in
    the input or output sequences [2, 19]. In all but a few cases [27], however,
    such attention mechanisms are used in conjunction with a recurrent network.
    In this work we propose the Transformer, a model architecture eschewing
    recurrence and instead relying entirely on an attention mechanism to draw
    global dependencies between input and output. The Transformer allows for
    significantly more parallelization and can reach a new state of the art in
    translation quality after being trained for as little as twelve hours on
    eight P100 GPUs. 
  example_title: Attention Is All You Need
- text: >-
    [Abstract] In this work, we explore prompt tuning, a simple yet effective
    mechanism for learning soft prompts to condition frozen language models to
    perform specific downstream tasks. Unlike the discrete text prompts used by
    GPT-3, soft prompts are learned through backpropagation and can be tuned to
    incorporate signal from any number of labeled examples. Our end-to-end
    learned approach outperforms GPT-3's few-shot learning by a large margin.
    More remarkably, through ablations on model size using T5, we show that
    prompt tuning becomes more competitive with scale: as models exceed billions
    of parameters, our method closes the gap and matches the strong performance
    of model tuning (where all model weights are tuned). This finding is
    especially relevant in that large models are costly to share and serve, and
    the ability to reuse one frozen model for multiple downstream tasks can ease
    this burden. Our method can be seen as a simplification of the recently
    proposed prefix tuning of Li and Liang (2021), and we provide a comparison
    to this and other similar approaches. Finally, we show that conditioning a
    frozen model with soft prompts confers benefits in robustness to domain
    transfer, as compared to full model tuning. [Introduction] With the wide
    success of pre-trained large language models, a range of techniques has
    arisen to adapt these general-purpose models to downstream tasks. ELMo
    (Peters et al., 2018) proposed freezing the pre-trained model and learning a
    task-specific weighting of its per-layer representations. However, since GPT
    (Radford et al., 2018) and BERT (Devlin et al., 2019), the dominant
    adaptation technique has been model tuning (or fine-tuning), where all model
    parameters are tuned during adaptation, as proposed by Howard and Ruder
    (2018).More recently, Brown et al. (2020) showed that prompt design (or
    priming) is surprisingly effective at modulating a frozen GPT-3 model’s
    behavior through text prompts. Prompts are typically composed of a task
    description and/or several canonical examples. This return to freezing
    pre-trained models is appealing, especially as model size continues to
    increase. Rather than requiring a separate copy of the model for each
    downstream task, a single generalist model can simultaneously serve many
    different tasks. Unfortunately, prompt-based adaptation has several key
    drawbacks. Task description is error-prone and requires human involvement,
    and the effectiveness of a prompt is limited by how much conditioning text
    can fit into the model’s input. As a result, downstream task quality still
    lags far behind that of tuned models. For instance, GPT-3 175B fewshot
    performance on SuperGLUE is 17.5 points below fine-tuned T5-XXL (Raffel et
    al., 2020) (71.8 vs. 89.3) despite using 16 times more parameters. Several
    efforts to automate prompt design have been recently proposed. Shin et al.
    (2020) propose a search algorithm over the discrete space of words, guided
    by the downstream application training data. While this technique
    outperforms manual prompt design, there is still a gap relative to model
    tuning. Li and Liang (2021) propose prefix tuning and show strong results on
    generative tasks. This method freezes the model parameters and
    backpropagates the error during tuning to prefix activations prepended to
    each layer in the encoder stack, including the input layer. Hambardzumyan et
    al. (2021) simplify this recipe by restricting the trainable parameters to
    the input and output subnetworks of a masked language model, and show
    reasonable results on classifications tasks. In this paper, we propose
    prompt tuning as a further simplification for adapting language models. We
    freeze the entire pre-trained model and only allow an additional k tunable
    tokens per downstream task to be prepended to the input text. This soft
    prompt is trained end-to-end and can condense the signal from a full labeled
    dataset, allowing our method to outperform few-shot prompts and close the
    quality gap with model tuning (Figure 1). At the same time, since a single
    pre-trained model is recycled for all downstream tasks, we retain the
    efficient serving benefits of frozen models (Figure 2). While we developed
    our method concurrently with Li and Liang (2021) and Hambardzumyan et al.
    (2021), we are the first to show that prompt tuning alone (with no
    intermediate-layer prefixes or task-specific output layers) is sufficient to
    be competitive with model tuning. Through detailed experiments in sections
    2–3, we demonstrate that language model capacity is a key ingredient for
    these approaches to succeed. As Figure 1 shows, prompt tuning becomes more
    competitive with scale. We compare with similar approaches in Section 4.
    Explicitly separating task-specific parameters from the generalist
    parameters needed for general language-understanding has a range of
    additional benefits. We show in Section 5 that by capturing the task
    definition in the prompt while keeping the generalist parameters fixed, we
    are able to achieve better resilience to domain shifts. In Section 6, we
    show that prompt ensembling, learning multiple prompts for the same task,
    can boost quality and is more efficient than classic model ensembling.
    Finally, in Section 7, we investigate the interpretability of our learned
    soft prompts. In sum, our key contributions are: 1. Proposing prompt tuning
    and showing its competitiveness with model tuning in the regime of large
    language models. 2. Ablating many design choices, and showing quality and
    robustness improve with scale. 3. Showing prompt tuning outperforms model
    tuning on domain shift problems. 4. Proposing prompt ensembling and showing
    its effectiveness. 
  example_title: PEFT (2104.08691)
- text: >-
    [Abstract] For the first time in the world, we succeeded in synthesizing the
    room-temperature superconductor (Tc≥400 K, 127∘C) working at ambient
    pressure with a modified lead-apatite (LK-99) structure. The
    superconductivity of LK-99 is proved with the Critical temperature (Tc),
    Zero-resistivity, Critical current (Ic), Critical magnetic field (Hc), and
    the Meissner effect. The superconductivity of LK-99 originates from minute
    structural distortion by a slight volume shrinkage (0.48 %), not by external
    factors such as temperature and pressure. The shrinkage is caused by Cu2+
    substitution of Pb2+(2) ions in the insulating network of Pb(2)-phosphate
    and it generates the stress. It concurrently transfers to Pb(1) of the
    cylindrical column resulting in distortion of the cylindrical column
    interface, which creates superconducting quantum wells (SQWs) in the
    interface. The heat capacity results indicated that the new model is
    suitable for explaining the superconductivity of LK-99. The unique structure
    of LK-99 that allows the minute distorted structure to be maintained in the
    interfaces is the most important factor that LK-99 maintains and exhibits
    superconductivity at room temperatures and ambient pressure. [Introduction] 
    Since the discovery of the first superconductor(1), many efforts to search
    for new roomtemperature superconductors have been carried out worldwide(2,
    3) through their experimental clarity or/and theoretical perspectives(4-8).
    The recent success of developing room-temperature superconductors with
    hydrogen sulfide(9) and yttrium super-hydride(10) has great attention
    worldwide, which is expected by strong electron-phonon coupling theory with
    high-frequency hydrogen phonon modes(11, 12). However, it is difficult to
    apply them to actual application devices in daily life because of the
    tremendously high pressure, and more efforts are being made to overcome the
    high-pressure problem(13). For the first time in the world, we report the
    success in synthesizing a room-temperature and ambient-pressure
    superconductor with a chemical approach to solve the temperature and
    pressure problem. We named the first room temperature and ambient pressure
    superconductor LK-99. The superconductivity of LK-99 proved with the
    Critical temperature (Tc), Zero-resistivity, Critical current (Ic), Critical
    magnetic field (Hc), and Meissner effect(14, 15). Several data were
    collected and analyzed in detail to figure out the puzzle of
    superconductivity of LK-99: X-ray diffraction (XRD), X-ray photoelectron
    spectroscopy (XPS), Electron Paramagnetic Resonance Spectroscopy (EPR), Heat
    Capacity, and Superconducting quantum interference device (SQUID) data.
    Henceforth in this paper, we will report and discuss our new findings
    including superconducting quantum wells associated with the
    superconductivity of LK-99.
  example_title: LK-99 (Not NLP)
- text: >-
    [Abstract] Abstract Evaluation practices in natural language generation
    (NLG) have many known flaws, but improved evaluation approaches are rarely
    widely adopted. This issue has become more urgent, since neural NLG models
    have improved to the point where they can often no longer be distinguished
    based on the surfacelevel features that older metrics rely on. This paper
    surveys the issues with human and automatic model evaluations and with
    commonly used datasets in NLG that have been pointed out over the past 20
    years. We summarize, categorize, and discuss how researchers have been
    addressing these issues and what their findings mean for the current state
    of model evaluations. Building on those insights, we lay out a long-term
    vision for NLG evaluation and propose concrete steps for researchers to
    improve their evaluation processes. Finally, we analyze 66 NLG papers from
    recent NLP conferences in how well they already follow these suggestions and
    identify which areas require more drastic changes to the status quo.
    [Introduction] There are many issues with the evaluation of models that
    generate natural language. For example, datasets are often constructed in a
    way that prevents measuring tail effects of robustness, and they almost
    exclusively cover English. Most automated metrics measure only similarity
    between model output and references instead of fine-grained quality aspects
    (and even that poorly). Human evaluations have a high variance and, due to
    insufficient documentation, rarely produce replicable results. These issues
    have become more urgent as the nature of models that generate language has
    changed without significant changes to how they are being evaluated. While
    evaluation methods can capture surface-level improvements in text generated
    by state-of-the-art models (such as increased fluency) to some extent, they
    are ill-suited to detect issues with the content of model outputs, for
    example if they are not attributable to input information. These ineffective
    evaluations lead to overestimates of model capabilities. Deeper analyses
    uncover that popular models fail even at simple tasks by taking shortcuts,
    overfitting, hallucinating, and not being in accordance with their
    communicative goals. Identifying these shortcomings, many recent papers
    critique evaluation techniques or propose new ones. But almost none of the
    suggestions are followed or new techniques used. There is an incentive
    mismatch between conducting high-quality evaluations and publishing new
    models or modeling techniques. While general-purpose evaluation techniques
    could lower the barrier of entry for incorporating evaluation advances into
    model development, their development requires resources that are hard to
    come by, including model outputs on validation and test sets or large
    quantities of human assessments of such outputs. Moreover, some issues, like
    the refinement of datasets, require iterative processes where many
    researchers collaborate. All this leads to a circular dependency where
    evaluations of generation models can be improved only if generation models
    use better evaluations. We find that there is a systemic difference between
    selecting the best model and characterizing how good this model really is.
    Current evaluation techniques focus on the first, while the second is
    required to detect crucial issues. More emphasis needs to be put on
    measuring and reporting model limitations, rather than focusing on producing
    the highest performance numbers. To that end, this paper surveys analyses
    and critiques of evaluation approaches (sections 3 and 4) and of commonly
    used NLG datasets (section 5). Drawing on their insights, we describe how
    researchers developing modeling techniques can help to improve and
    subsequently benefit from better evaluations with methods available today
    (section 6). Expanding on existing work on model documentation and formal
    evaluation processes (Mitchell et al., 2019; Ribeiro et al., 2020), we
    propose releasing evaluation reports which focus on demonstrating NLG model
    shortcomings using evaluation suites. These reports should apply a
    complementary set of automatic metrics, include rigorous human evaluations,
    and be accompanied by data releases that allow for re-analysis with improved
    metrics. In an analysis of 66 recent EMNLP, INLG, and ACL papers along 29
    dimensions related to our suggestions (section 7), we find that the first
    steps toward an improved evaluation are already frequently taken at an
    average rate of 27%. The analysis uncovers the dimensions that require more
    drastic changes in the NLG community. For example, 84% of papers already
    report results on multiple datasets and more than 28% point out issues in
    them, but we found only a single paper that contributed to the dataset
    documentation, leaving future researchers to re-identify those issues. We
    further highlight typical unsupported claims and a need for more consistent
    data release practices. Following the suggestions and results, we discuss
    how incorporating the suggestions can improve evaluation research, how the
    suggestions differ from similar ones made for NLU, and how better metrics
    can benefit model development itself (section 8). 
  example_title: NLG-Eval (2202.06935)
model-index:
- name: Long-paper-summarization-pegasus-x-b
  results:
  - task:
      name: Summarization
      type: summarization
    dataset:
      name: ccdv/arxiv-summarization
      type: ccdv/arxiv-summarization
      config: section
      split: test
      args: section
    metrics:
    - name: ROUGE-1
      type: rouge
      value: 35.6639
    - name: ROUGE-2
      type: rouge
      value: 9.81362
    - name: ROUGE-L
      type: rouge
      value: 19.9013      
    - name: ROUGE-LSum
      type: rouge
      value: 28.1444
    
license: mit
language:
- en
metrics:
- rouge

---




<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Long-paper-summarization-pegasus-x-b

This model is a fine-tuned version of [google/pegasus-x-base](https://huggingface.co/google/pegasus-x-base) on the arxiv-summarization dataset.
It achieves the following results on the evaluation set:
- Loss: 2.7262


## Model Description / Training and evaluation data


**Base Model**: [Pegasus-x-base (State-of-the-art for Long Context Summarization)](https://huggingface.co/google/pegasus-x-base)

**Finetuning Dataset**: 
- We used **train[25000:100000] of ArXiv Dataset (Cohan et al., 2018, NAACL-HLT 2018)** [[PDF]](https://arxiv.org/abs/1804.05685)
- (Full length is 200,000+, We will upload full trained Model soon)

**GPU**: (RTX A6000) x 1

**Train time**: About 24 hours for 3 epochs

**Test time**: About 8 hours for test dataset.


## Intended uses & limitations

- **Research Paper Summarization**



### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 64
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 390
- **num_epochs: 3 (takes about 24 hours)**

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 3.401         | 0.33  | 390  | 2.3985          |
| 2.5444        | 0.67  | 780  | 2.2461          |
| 2.4849        | 1.0   | 1170 | 2.2690          |
| 2.5735        | 1.33  | 1560 | 2.3334          |
| 2.7045        | 1.66  | 1950 | 2.4330          |
| 2.8939        | 2.0   | 2340 | 2.5461          |
| 3.0773        | 2.33  | 2730 | 2.6502          |
| 3.2149        | 2.66  | 3120 | 2.7039          |
| 3.2844        | 3.0   | 3510 | 2.7262          |



### Framework versions

- Transformers 4.32.1
- Pytorch 2.0.1
- Datasets 2.12.0
- Tokenizers 0.13.2