BabakScrapes/disco-clause-segmenter

DiSCo Clause Segmenter

A roberta-base token-classification model that splits English text into Elementary Discourse Unites, roughly clauses. Each token is tagged with a 3-way label and clause spans are recovered by matching the pattern B I* E:

id	label	meaning
0	B	clause beginning
1	I	inside clause
2	E	clause end

This is the segmentation component of the DiSCo pipeline; feed its clause output to the companion BabakScrapes/disco-se-classifier for Situation Entity typing.

Usage

from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch

tok = AutoTokenizer.from_pretrained("roberta-base", add_prefix_space=True)
model = AutoModelForTokenClassification.from_pretrained("BabakScrapes/disco-clause-segmenter").eval()

words = "There was bad weather at the airport and so our flight got delayed".split()
enc = tok(words, is_split_into_words=True, return_tensors="pt", truncation=True, max_length=512)
with torch.inference_mode():
    token_preds = model(**enc).logits.argmax(-1)[0].tolist()
# align token_preds back to words (majority vote per word), then cut clauses on B...E

See pipeline.py in the DiSCo release for the full word-alignment + B I* E clause-recovery logic.

Training data

Trained on the public SitEnt corpus (Friedrich et al. 2016/2017), converted to B/I/E token labels, with a deterministic split (seed 42, 90/10 train/validation). The construction recipe (construct_clause_corpus.ipynb) is distributed with the DiSCo code/corpus release, not in this model repo.

Performance

In-domain SitEnt held-out evaluation (seed-42 10% partition; 16,515 tokens, 1,848 gold clause spans):

Metric	Value
Token macro-F1	.753 (B .703 / I .917 / E .639)
Token accuracy (= micro-F1)	.861
Gold clauses with ≥50% predicted-overlap coverage	95.5%
Predicted vs gold clause counts	2,149 vs 1,848 (slight over-segmentation)

The segmenter recovers clause regions reliably; the span-overlap numbers are the operationally relevant summaries because the downstream pipeline consumes whole clauses rather than exact boundary tokens. Macro-F1 is depressed by the sparse B/E classes.

Limitations

• English only.
• Tends to slightly over-segment, which is benign for downstream SE classification.

Citation

Hemmatian, B. (2022). Taking the High Road: A Big Data Investigation of Natural Discourse in the Emerging U.S. Consensus about Marijuana Legalization. Brown University. And the DiSCo corpus paper (forthcoming, Behavior Research Methods). SitEnt: Friedrich, Palmer & Pinkal (2016).