NicolasSperanza/hw1-hc3-detector

HC3 AI-Generated Text Detector

For this assignment, I used the HC3 dataset to train a model to classify answers as either human-written or ChatGPT-generated. I used sentence-transformers/all-MiniLM-L6-v2 as the base model, with 0 representing human answers and 1 representing ChatGPT answers. The dataset was split into training, validation, and testing using an 80/10/10 split with seed 42, and answers to the same question were kept in the same split.

I first created a baseline by keeping the MiniLM model frozen, generating sentence embeddings, and training a logistic regression classifier on those embeddings. The baseline reached a test accuracy of 0.8449. It correctly classified 1946 human answers and 1998 ChatGPT answers, while 388 human answers were classified as ChatGPT and 336 ChatGPT answers were classified as human.

After that, I fine-tuned the transformer directly for the classification task. I trained it for 5 epochs using AdamW with a learning rate of 2e-5, a batch size of 32, and a maximum sequence length of 256. The fine-tuned model reached a test accuracy of 0.9942, with 2308 human answers and 2333 ChatGPT answers classified correctly. Only 26 human answers were classified as ChatGPT and 1 ChatGPT answer was classified as human, bringing the total number of mistakes down from 724 with the baseline to 27 after fine-tuning.

Even though the final accuracy is really high, I wouldn't consider this a perfect AI detector. Since the model was trained and tested on HC3, it could be learning patterns that are specific to this dataset, like differences in answer length, structure, or writing style. Because of that, the same model might not perform as well on newer AI models or completely different types of writing.