Package: bibliograph
bibliographA database system for research in the humanities.bibliographis a Python package that implements a unique data model designed to help researchers analyze complex source material. The database was designed for academic studies in history, where researchers are often required to manage large amounts of ambiguous, inconsistent, or potentially contradictory information from a variety of sources.I am testing the beta and I will document the code when I've gotten everything running for my current use case.Overview: picking your mess"Data cleaning" typically destroys information: data are restructured such that references are uniform, duplicates are eliminated, and the database contains a "correct" representation of input data. In that process, a database is irreversibly separated from whatever sources generated the input data. The standard assumption is that inconsistencies in data are either entry errors or useless variations on a standard value. However, those variations between statements about the same things are exactly the kind of data historians and other researchers in the humanities would like to study.bibliographretains the variations alongside normalized, queryable tables so users can access both the "clean" relationships they interpret from source material and the "messy" relationships in their sources.As a simple example, consider a data set with assertions that "Miss Gerould", "Malkus, J.", "Joanne Simpson", "Gerould, Joanne", "Mrs. Malkus", and "simpsonj" represent contributors to scientific papers. These are all names of the same meteorologist. In thebibliographschema, these string values are stored in one table and another table contains links between them and strings representing the papers. A third table has a single row representing one person and all the string values reference that row, which I call a node. A fourth table then stores links between nodes, so the node for Simpson could be linked to nodes representing articles she wrote.Critically, with a large data set, the set of assertions about strings that map to the same node will be inconsistent and include false statements. Simpson could be listed in the wrong position in an author list, she could be missing from an author list, have the wrong affiliation, etc. In abibliographdatabase, all those inconsistent or incorrect relations are retained and the user can add, change, or remove links between nodes as-needed to reflect their preferred relations. This schema allows a researcher to easily ask questions about sets of sources:Which sources referred to Simpson as "Miss" or "Mrs." with a surname instead of using her full name?Did she use different names in different contexts?Did some journals or institutions require certain names?"Clean" data often can't be used to answer those kinds of questions.bibliographasks a user to pick out a mess they can use to solve research problems.The data modelMost implications of the data model are contained in the sentenceInput data consist of assertions about strings, which correspond to edges between nodesAs show in the diagram below, the database is built around four tables:assertions,strings,nodes, andedges.All text values are stored in thestringstableTheassertionstable contains links between text values in sources: if a document says "A is author of B", a user might store "A" and "B" in thestringstable and create a link of type "author" in theassertionstable.Each string is mapped to a row in thenodestable which represents some entity outside the database: "A" above could get mapped to a node of type "person" and "B" could be mapped to a node of type "article."Rows in theedgestable define links between nodes: if the source document above was incorrect and "A" was in fact the editor of "B", a user could create a link of type "editor" in theedgestable.The database exposes normalized relationships the user considers valid and simultaneously retains an accurate description of all source material. Theassertionsandstringstables represent statements in sources which could be valid or invalid andbibliographhelps a user interpret those statements as relationships between abstractnodesandedges.Two things to note about the description above:None of the example associations are hard-coded. A user can have "author" as a node type rather than a link type, the node types could be "actor" and "work", "A is author of B" could be stored twice in thestringstable with each instance mapped to a different node, etc. The only constraint is that strings are mapped to nodes many-to-one: every row in thestringstable points to only one node but any number of strings can refer to the same row in thenodestable.No database can prevent bad input (all the statements about sources here assume a user has created input data that accurately describe their sources), butbibliographallows a user to search for and resolve errors in ways they couldn't if data were cleaned before entry: when an error is suspected, users can compare the abstract relationships they think are correct with "raw" input values and more easily distinguish between user error and ambiguous sources. Rows in theassertionsandedgestables contain optional metadata that allows users to quickly determine how to compare their input data to external sources. See the (forthcoming) documentation for details.