HW1 HC3 Detector

Binary English answer classifier for CS546 HW1, fine-tuned from sentence-transformers/all-MiniLM-L6-v2. Labels: 0 = human, 1 = ChatGPT. Inputs are answer text only, truncated to 256 tokens.

Measured results

Model	Test accuracy	Macro precision	Macro recall	Macro F1
Frozen sentence embeddings + logistic regression	0.844901	0.845073	0.844901	0.844882
Fine-tuned classifier	0.985219	0.985643	0.985219	0.985215

Confusion matrices (rows = true labels, columns = predicted labels; human, ChatGPT):

Baseline: [[1946, 388], [336, 1998]]
Fine-tuned: [[2265, 69], [0, 2334]]

Training and evaluation

HC3 English revision 4d0ff18143b5a7e1b1e79beb540c04549d1e59d3; first nonempty answer from each class per question. The supplied preparation excludes empty questions, missing or identical answer pairs, and repeated questions. Normalized questions are split 80/10/10 with seed 42 before flattening answers. Sizes: {'train': 37334, 'validation': 4666, 'test': 4668}. No question occurs in more than one split. Base-model revision: 1110a243fdf4706b3f48f1d95db1a4f5529b4d41.

All transformer parameters were fine-tuned with AdamW (learning rate 2e-05, weight decay 0.01), batch size 32, maximum length 256, and 5 complete epochs, using seed 42. Validation was monitored after each epoch. This upload is the final fifth-epoch model; the test set was not used to select checkpoints or hyperparameters. Mean batch training loss: 0.022178. Environment: PyTorch 2.8.0, Transformers 4.57.1, SentenceTransformers 5.1.2, Datasets 4.1.1, scikit-learn 1.7.2; device: mps. Full measured metrics are in evaluation_results.json.

Limitations

This is an educational classifier evaluated on a historical, balanced HC3 benchmark. It may learn dataset-specific style, source, or length cues and is not validated on current language models, other languages, or other domains. False positives are possible; it must not be used as reliable evidence that a student used AI.

Dataset: Guo et al. (2023), How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection, https://arxiv.org/abs/2301.07597.