GitHub repository: onesuper/HuggingFace-Datasets-Text-Quality-Analysis
Primary language: Python
Created: 2023-06-16T14:51:48Z
Description: Retrieves parquet files from Hugging Face, identifies and quantifies junky data, duplication, contamination, and biased content in dataset using pandas