Package: corpusshow
Corpus-ShowCorpus-Show helps to understand the corpus data distribution through various values generated from Sentence Transformer. (It's not such a great package, but It simply helps you visualize comfortably.)Corpus-Show performs sentence embedding via Sentence Transformers, a Python framework for state-of-the-art sentence, text and image embeddings.[Paper][Document][Huggingface model]You can visualize the embedded sentences of each document generated from SentenceTransformers.Corpus-Show can also generate clusters with sentences embedded array throughScikit-Learn KMeans.The sentence transformer model is downloaded through the hugging face interface, and the default model is set toparaphrase-xlm-r-multilingual-v1, which supports multiple languages. However, you can easily input your custom model as a sentence transformer model through the hugging face interface. It is also easy to fine-tune via SBERT. For more models, please seethis page.InstallationpipinstallcorpusshowThis package may not work properly in m1/m2 mac environment. If you are using Mac m1/m2, please use the git repository as a submodule because it has minimal encapsulation.[issue#1]TutorialWe provide tutorial notebooks for all the features we offer. We plan to provide additional docstrings or documentation from the official distribution version (major version 1 or higher).Main-tutorials:https://github.com/DSDanielPark/corpus-show/blob/main/tutorials/corpusshow_tutorial.ipynbSub-tutorial-folder:https://github.com/DSDanielPark/corpus-show/blob/main/tutorialsMain FeatureIt helps to create a simple but useful plot as shown below with a simple dataframe and column names as input, such as the followingBBC sample datasetin./data/bbc_news_dataset.csv.newstopic0Oil rebounds from weather effect (...)business1Indonesia 'declines debt freeze' (...)business.........601EU software patent law faces axe (...)tech1.CorpusClsterContains 1 static method. You can create great pictures with:fromcorpusshowimportCorpusCluster# Class argumentscsv_file_path='../data/bbc_news_dataset.csv'sentence_transformer_model_name='paraphrase-xlm-r-multilingual-v1'target_col='news'num_cluster=4# Get class objectcc=CorpusCluster(csv_file_path,sentence_transformer_model_name,target_col,num_cluster)# 1. quick_corpus_show method:# Show figures without k-means clusteringcc.quick_corpus_show('topic','tsne2d',False,'fig1.png')cc.quick_corpus_show('topic','tsne3d',False,'fig2.png')cc.quick_corpus_show('topic','pca2d',False,'fig3.png')cc.quick_corpus_show('topic','pca3d',False,'fig4.png')# 2. quick_cluster_show method:# Show figures with k-means clusteringdf_returned=cc.quick_cluster_show('tsne2d',False,'fig5.png')df_returned=cc.quick_cluster_show('tsne3d',False,'fig6.png')df_returned=cc.quick_cluster_show('pcda2d',False,'fig7.png')df_returned=cc.quick_cluster_show('pcda2d',False,'fig8.png')If you want to change the design of the plot, use matplotlib's RcParams method or the returned dataframe.References[1] Scikit-Learnhttps://scikit-learn.org[2] Matplotlibhttps://matplotlib.org/[3] Huggingface Sentence Transformerhttps://huggingface.co/sentence-transformers[4] SBERThttps://www.sbert.net/Use Case[1]Korean-news-topic-classification-using-KO-BERT: all plots were created through Corpus-Show and Quick-Show.ContactsMaintainer:Daniel Park, South Koreae-mailparkminwoo1991@gmail.com