Package: turftopic
Topic modeling is your turf too.Contextual topic models with representations from transformers.IntentionsProvide simple, robust and fast implementations of existing approaches (BERTopic, Top2Vec, CTM) with minimal dependencies.Implement state-of-the-art approaches from my papers. (papers work-in-progress)Put all approaches in a broader conceptual framework.Provide clear and extensive documentation about the best use-cases for each model.Make the models' API streamlined and compatible with topicwizard and scikit-learn.Develop smarter, transformer-based evaluation metrics.!!!This package is still a prototype, and no papers are published about the models. Until these are out, and most features are implemented
I DO NOT recommend using this package for production and academic use!!!RoadmapModel ImplementationPretty PrintingImplement visualization utilites for these models in topicwizardThorough documentationPublish papers :hourglass_flowing_sand: (in progress..)Dynamic modeling :hourglass_flowing_sand:More TutorialsHigh-level topic descriptions with LLMs.Contextualized evaluation metrics.Basics(Documentation)InstallationTurftopic can be installed from PyPI.pipinstallturftopicIf you intend to use CTMs, make sure to install the package with Pyro as an optional dependency.pipinstallturftopic[pyro-ppl]Fitting a ModelTurftopic's models follow the scikit-learn API conventions, and as such they are quite easy to use if you are familiar with
scikit-learn workflows.Here's an example of how you use KeyNMF, one of our models on the 20Newsgroups dataset from scikit-learn.fromsklearn.datasetsimportfetch_20newsgroupsnewsgroups=fetch_20newsgroups(subset="all",remove=("headers","footers","quotes"),)corpus=newsgroups.dataTurftopic also comes with interpretation tools that make it easy to display and understand your results.fromturftopicimportKeyNMFmodel=KeyNMF(20).fit(corpus)Interpreting ModelsTurftopic comes with a number of pretty printing utilities for interpreting the models.To see the highest the most important words for each topic, use theprint_topics()method.model.print_topics()Topic IDTop 10 Words0armenians, armenian, armenia, turks, turkish, genocide, azerbaijan, soviet, turkey, azerbaijani1sale, price, shipping, offer, sell, prices, interested, 00, games, selling2christians, christian, bible, christianity, church, god, scripture, faith, jesus, sin3encryption, chip, clipper, nsa, security, secure, privacy, encrypted, crypto, cryptography....# Print highest ranking documents for topic 0model.print_highest_ranking_documents(0,corpus,document_topic_matrix)DocumentScorePoor 'Poly'. I see you're preparing the groundwork for yet another retreat from your...0.40Then you must be living in an alternate universe. Where were they? An Appeal to Mankind During the...0.40It is 'Serdar', 'kocaoglan'. Just love it. Well, it could be your head wasn't screwed on just right...0.39model.print_topic_distribution("I think guns should definitely banned from all public institutions, such as schools.")Topic nameScore7_gun_guns_firearms_weapons0.0517_mail_address_email_send0.003_encryption_chip_clipper_nsa0.0019_baseball_pitching_pitcher_hitter0.0011_graphics_software_program_3d0.00VisualizationTurftopic does not come with built-in visualization utilities,topicwizard, an interactive topic model visualization library, is compatible with all models from Turftopic.pipinstalltopic-wizardBy far the easiest way to visualize your models for interpretation is to launch the topicwizard web app.importtopicwizardtopicwizard.visualize(corpus,model=model)Screenshot of the topicwizard Web ApplicationAlternatively you can use theFigures APIin topicwizard for individual HTML figures.ModelsModelDescriptionUsageKeyNMFNon-negative Matrix Factorization enhanced with keyword extraction using sentence embeddingsmodel = KeyNMF(n_components=10).fit(corpus)GMMGaussian Mixture Model over contextual embeddings + post-hoc term importance estimationmodel = GMM(n_components=10).fit(corpus)S³Separates semantic signals, aka. axes of semantics in a corpus using independent component analysis.model = SemanticSignalSeparation(n_components=10).fit(corpus)Autoencoding ModelsLearn topics using amortized variational inference enhanced by contextual representations.model = AutoEncodingTopicModel(n_components=10, combined=False).fit(corpus)Clustering ModelsClusters semantic embeddings, and estimates term importances for clusters.model = ClusteringTopicModel(feature_importance="ctfidf").fit(corpus)For extensive comparison see ourModel Overview.