Package: sashimi-domains
Sashimi - study the organisation and evolution of corporaSashimi is a Python module that provides tailored mathematical models and corresponding interactive visualisations
for exploratory and confirmatory mixed-methods analysis of large textual or token corpora. It can detect textual
and metadata structures and shifts, by employing stochastic block modeling (SBM) fromgraph-toolor
[currently deprecated] word embedding fromGensim.Models:Domain-topic modelsDomain-chained (metadata) modelsModel-based data interfaces (visualisations):Interactive domain-topic maps and domain-chained mapsDomain-topic tables and domain-chained tablesDomain-topic, domain-chained and domain-topic-chained networksArea rank charts (bump charts) of the evolution of domain sizesUsing Sashimi without programming (no code)Sashimi is available as a suite of methods in theCortext Managerweb service. SeeSASHIMI.Savoring SashimiAlso for users of this library, thedocumentationat Cortext serves as a good introduction to the methodology.InstallationInstallgraph-toolaccording to your system.Then:pip install sashimi-domainsDependenciesThis project builds mainly on the following others:graph-tool(for Stochastic Block Model inference)pandasspacy(for tokenization)gensim(for ngram detection)bokeh(for building hierarchical block maps and other plots)lxml(for building domain tables)matplotlib(for plotting simple corpus statistics)With the exception of graph-tool, they'll be automatically handled bypip.Basic usagefromsashimiimportGraphModelsimportpandasaspd# Let's instantiate a corpus with an explicit storage dircorpus=GraphModels(storage_dir="my_project")# Read a dataframe from a CSV file and load it into the corpus, giving it a namedf=pd.read_csv("example_corpus.csv")corpus.load_data(df,name='example')# Take a look at what was loadedprint(corpus.data)# For autoloading, store the data in JSON format under the project's storage dircorpus.store_data()Preparing the dataTextual dataFor the typical usage with a textual corpus:# Set the data column labels for text sourcescorpus.text_sources=["title","body"]# Set up how to process text sources, for example:text_sources_args=dict(ngrams=3,language="en",stop_words=True)Thelanguageis any validspacylanguage, whereNonewill use the English tokenizer with no stop_words. Our English tokenizer is slightly improved from spacy's original, in order to get ["hot-dog"] out of "hot-dog" (when spacy would split that), ["this", "that"] out of "this/that", and ["citation"] out of "citation[2,3]".Token dataYou may also directly use tokens that you've processed yourself, or from token data such as keywords, categories, or anything really, in addition or in place of textual data.# Set the data column labels for token sourcescorpus.token_sources=["keyword","category"]Token data for a document is expected to be in the form of a list, containing strings or lists containing strings:List[str | List[str]]. If your data differs, you must adjust it before processing.Processing the dataOnce token and text sources are set up, you can process them all with:corpus.process_sources(**text_sources_args)Working with your corpus# Set data column labelscorpus.col_title="titles"# document titles (required; if your corpus doesn't have canonical titles, be creative)corpus.col_time="years"# dates (optional)corpus.col_urls="urls"# may also be a list of columns, for multiple urls (optional)# Load a domain-topic modelcorpus.load_domain_topic_model()print(corpus.state)print(corpus.dblocks)print(corpus.tblocks)# Create an interactive hierarchical map# Output is a self-contained html+css+js+data documentcorpus.domain_map()# Create network representations for the first domain and topic levels, as pdf, svg and graphml documentscorpus.domain_network(doc_level=1,ter_level=1)corpus.domain_network(doc_level=2,ter_level=1)# The default representation shows edges from a domain to its "specific" topics, with the previous call being equivalent to the followingcorpus.domain_network(doc_level=2,ter_level=1,edges="specific")# For domains of level higher than 1, setting `edges="common"` will show edges towards topics that are common to all of its subdomainscorpus.domain_network(doc_level=2,ter_level=1,edges="common")# Create domain-topic tables for all domains at level 3if3incorpus.dblocks:corpus.subxblocks_tables(xbtype="doc",xlevel=3,xb=None,ybtype="ter")# Store the current choices of corpus and modelcorpus.register_config("my_config.json")# To reload them in a future sessionfromsashimiimportGraphModelscorpus=GraphModels('my_config.json')corpus.load_domain_topic_model()# will load the previously calculated modelcorpus.load_domain_topic_model(load=False)# will fit a new modelprint(corpus.list_blockstates())# Load a domain-chained model over the column "metadata_A"corpus.set_chain(prop='metadata_A'}corpus.load_domain_chained_model()print(corpus.list_chainedbstates())# Create interactive instruments for the chained dimensioncorpus.domain_map(chained=True)corpus.domain_network(doc_level=1,ter_level=None,ext_level=1)corpus.domain_network(doc_level=1,ter_level=1,ext_level=1)# The `edges` parameter applies to both topics and the chained dimensioncorpus.domain_network(doc_level=2,ter_level=None,ext_level=1,edges="specific")corpus.domain_network(doc_level=2,ter_level=1,ext_level=1,edges="common")if3incorpus.dblocks:corpus.subxblocks_tables(xbtype="doc",xlevel=3,xb=None,ybtype="ext")Advanced usagesThe domain map document provides a "Help" tab explaining how to navigate and read it, which is also useful to understand the other interfaces.Create filtered visualisations to show only a selected group of domains in maps and networks.Perform selective chaining, whereby a domain-chanied model is fit by considering only a selected group of domains, instead of to the entire corpus, in order to understand the local insertion of metadata dimensions.Calls toCorpus.set_chainmay pass in thematcherparameter a path to a json file containing a dictionary. Nodes of the chained dimension will then correspond to the keys in the dictionary, and links will be established by searching for a key's value, as a regular expression, in the column passed as the first parameter.DevelopmentThis module provides four main classes:class GraphModels(user-facing class, inherits Corpus and Blocks)Provides Stochastic Block Models of corpora from their document-term and document-metadata graphs, yielding domain-topic and domain-chained models, respectively.class BlocksProvides interactive domain maps, networks, tables, and other interfaces.class CorpusProvides loading and preprocessing corpora, plus some descriptive statistics.class Vectorology[currently deprecated]Provides models of corpora using word embedding and produces reports, statistics and visualisations.