LEDBill / README.md
d0r1h's picture
Librarian Bot: Add base_model information to model (#4)
1da3120 verified
metadata
license: apache-2.0
tags:
  - summarization
datasets: billsum
widget:
  - text: >-
      The people of the State of California do enact as follows: SECTIONHEADER
      Section 1170.02 is added to the Penal Code, to read: 1170.02. A prisoner
      is not eligible for resentence or recall pursuant to subdivision (e) of
      Section 1170 if he or she was convicted of first-degree murder if the
      victim was a peace officer, as defined in Section 830.1, 830.2, 830.3,
      830.31, 830.32, 830.33, 830.34, 830.35, 830.36, 830.37, 830.4, 830.5,
      830.6, 830.10, 830.11, or 830.12, who was killed while engaged in the
      performance of his or her duties, and the individual knew, or reasonably
      should have known, that the victim was a peace officer engaged in the
      performance of his or her duties, or the victim was a peace officer or a
      former peace officer under any of the above-enumerated sections, and was
      intentionally killed in retaliation for the performance of his or her
      official duties. SECTIONHEADER Section 3550 of the Penal Code is amended
      to read: 3550. Notwithstanding any other law, except as provided in
      subdivision (b), if the head physician of an institution in which a
      prisoner is incarcerated determines, as provided in this section, that the
      prisoner is permanently medically incapacitated with a medical condition
      that renders him or her permanently unable to perform activities of basic
      daily living, and results in the prisoner requiring 24-hour care, and that
      incapacitation did not exist at the time of sentencing, the prisoner shall
      be granted medical parole if the Board of Parole Hearings determines that
      the conditions under which he or she would be released would not
      reasonably pose a threat to public safety. This section does not alter or
      diminish the rights conferred under the Victims Bill of Rights Act of 2008
      . Subdivision (a) does not apply to any of the following: A prisoner
      sentenced to death or life in prison without possibility of parole. A
      prisoner who is serving a sentence for which parole, pursuant to
      subdivision (a), is prohibited by any initiative statute. A prisoner who
      was convicted of first-degree murder if the victim was a peace officer, as
      defined in Section 830.1, 830.2, 830.3, 830.31, 830.32, 830.33, 830.34,
      830.35, 830.36, 830.37, 830.4, 830.5, 830.6, 830.10, 830.11, or 830.12,
      who was killed while engaged in the performance of his or her duties, and
      the individual knew, or reasonably should have known, that the victim was
      a peace officer engaged in the performance of his or her duties, or the
      victim was a peace officer or a former peace officer under any of the
      above-enumerated sections, and was intentionally killed in retaliation for
      the performance of his or her official duties. When a physician employed
      by the Department of Corrections and Rehabilitation who is the primary
      care provider for a prisoner identifies a prisoner that he or she believes
      meets the medical criteria for medical parole specified in subdivision
      (a), the primary care physician shall recommend to the head physician of
      the institution where the prisoner is located that the prisoner be
      referred to the Board of Parole Hearings for consideration for medical
      parole. Within 30 days of receiving that recommendation, if the head
      physician of the institution concurs in the recommendation of the primary
      care physician, he or she shall refer the matter to the Board of Parole
      Hearings using a standardized form and format developed by the department,
      and if the head physician of the institution does not concur in the
      recommendation, he or she shall provide the primary care physician with a
      written explanation of the reasons for denying the referral.
      Notwithstanding any other provisions of this section, the prisoner or his
      or her family member or designee may independently request consideration
      for medical parole by contacting the head physician at the prison or the
      department. Within 30 days of receiving the request, the head physician of
      the institution shall, in consultation with the prisoners primary care
      physician, make a determination regarding whether the prisoner meets the
      criteria for medical parole as specified in subdivision (a) and, if the
      head physician of the institution determines that the prisoner satisfies
      the criteria set forth in subdivision (a), he or she shall refer the
      matter to the Board of Parole Hearings using a standardized form and
      format developed by the department. If the head physician of the
      institution does not concur in the recommendation, he or she shall provide
      the prisoner or his or her family member or designee with a written
      explanation of the reasons for denying the application. The Department of
      Corrections and Rehabilitation shall complete parole plans for inmates
      referred to the Board of Parole Hearings for medical parole consideration.
      The parole plans shall include, but not be limited to, the inmates plan
      for residency and medical care. Notwithstanding any other law, medical
      parole hearings shall be conducted by two-person panels consisting of at
      least one commissioner. In the event of a tie vote, the matter shall be
      referred to the full board for a decision. Medical parole hearings may be
      heard in absentia. Upon receiving a recommendation from the head physician
      of the institution where a prisoner is located for the prisoner to be
      granted medical parole pursuant to subdivision (c) or (d), the board, as
      specified in subdivision (f), shall make an independent judgment regarding
      whether the conditions under which the inmate would be released pose a
      reasonable threat to public safety, and make written findings related
      thereto. Notwithstanding any other law, the board or the Division of Adult
      Parole Operations shall have the authority to impose any reasonable
      conditions on prisoners subject to medical parole supervision pursuant to
      subdivision (a), including, but not limited to, the requirement that the
      parolee submit to electronic monitoring. As a further condition of medical
      parole, pursuant to subdivision (a), the parolee may be required to submit
      to an examination by a physician selected by the board for the purpose of
      diagnosing the parolees current medical condition. In the event such an
      examination takes place, a report of the examination and diagnosis shall
      be submitted to the board by the examining physician. If the board
      determines, based on that medical examination, that the persons medical
      condition has improved to the extent that the person no longer qualifies
      for medical parole, the board shall return the person to the custody of
      the department. Notwithstanding any other law establishing maximum periods
      for parole, a prisoner sentenced to a determinate term who is placed on
      medical parole supervision prior to the earliest possible release date and
      who remains eligible for medical parole, shall remain on medical parole,
      pursuant to subdivision (a), until that earliest possible release date, at
      which time the parolee shall commence serving that period of parole
      provided by, and under the provisions of, Chapter 8 of Title 1.
      Notwithstanding any other law establishing maximum periods for parole, a
      prisoner sentenced to an indeterminate term who is placed on medical
      parole supervision prior to the prisoners minimum eligible parole date,
      and who remains eligible for medical parole, shall remain on medical
      parole pursuant to subdivision (a) until that minimum eligible parole
      date, at which time the parolee shall be eligible for parole consideration
      under all other provisions of Chapter 8 of Title 1. The Department of
      Corrections and Rehabilitation shall, at the time a prisoner is placed on
      medical parole supervision pursuant to subdivision (a), ensure that the
      prisoner has applied for any federal entitlement programs for which the
      prisoner is eligible, and has in his or her possession a discharge medical
      summary, full medical records, parole medications, and all property
      belonging to the prisoner that was under the control of the department.
      Any additional records shall be sent to the prisoners forwarding address
      after release to health care-related parole supervision. The provisions
      for medical parole set forth in this title shall not affect an inmates
      eligibility for any other form of parole or release provided by law. (1)
      Notwithstanding any other law, the Department of Corrections and
      Rehabilitation shall give notice to the county of commitment and the
      proposed county of release, if that county is different than the county of
      commitment, of any medical parole hearing as described in subdivision (f),
      and of any medical parole release as described in subdivision (g). Notice
      shall be made at least 30 days, or as soon as feasible, prior to the time
      any medical parole hearing or medical parole release is scheduled for an
      inmate receiving medical parole consideration, regardless of whether the
      inmate is sentenced either determinately or indeterminately.
base_model: led-base-16384
model-index:
  - name: d0r1h/LEDBill
    results:
      - task:
          type: summarization
          name: Summarization
        dataset:
          name: billsum
          type: billsum
          config: default
          split: test
        metrics:
          - type: rouge
            value: 38.6502
            name: ROUGE-1
            verified: true
          - type: rouge
            value: 18.5458
            name: ROUGE-2
            verified: true
          - type: rouge
            value: 25.6561
            name: ROUGE-L
            verified: true
          - type: rouge
            value: 33.1575
            name: ROUGE-LSUM
            verified: true
          - type: loss
            value: 2.1305277347564697
            name: loss
            verified: true
          - type: gen_len
            value: 288.372
            name: gen_len
            verified: true

Longformer Encoder-Decoder (LED) fine-tuned on Billsum

This model is a fine-tuned version of led-base-16384 on the billsum dataset.

As described in Longformer: The Long-Document Transformer by Iz Beltagy, Matthew E. Peters, Arman Cohan, led-base-16384 was initialized from bart-base since both models share the exact same architecture. To be able to process 16K tokens, bart-base's position embedding matrix was simply copied 16 times.

How to use

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("d0r1h/LEDBill")
model = AutoModelForSeq2SeqLM.from_pretrained("d0r1h/LEDBill", return_dict_in_generate=True).to(device)

case = "......."

input_ids = tokenizer(case, return_tensors="pt").input_ids.to(device)
global_attention_mask = torch.zeros_like(input_ids)
global_attention_mask[:, 0] = 1

sequences = model.generate(input_ids, 
                           global_attention_mask=global_attention_mask).sequences
summary = tokenizer.batch_decode(sequences, 
                                 skip_special_tokens=True)
                                 

Evaluation results

When the model is used for summarizing Billsum documents(10 sample), it achieves the following results:

Model rouge1-f rouge1-p rouge2-f rouge2-p rougeL-f rougeL-p
LEDBill 34 37 15 16 30 32
led-base 2 15 0 0 2 15

This notebook shows how led can effectively be used for downstream task such summarization.