Package: dphon
dphoninstallationthis software is tested on the latest versions of macOS, windows, and ubuntu. you will need a supported version of python (above), along withpip.$pipinstalldphonif you're on windows and are seeing incorrectly formatted output in your terminal, have a look at thisstackoverflow answer.usagebasicsthe main function ofdphonis to look for instances of text reuse in a corpus of old chinese texts. instead of relying purely on graphemes, it does this by performing grapheme-to-phoneme conversion, and determining possible reuse based on whether passages are likely to havesoundedsimilar (or rhymed) when spoken aloud.you will need to have files stored locally as utf-8 encoded plain-text (.txt) or json-lines (.jsonl) format. for the former, one file is assumed to represent one document. for the latter, one file can contain any number of lines, each of which is a document, with required keysid(a unique identifier) andtext(text content) and any number of optional keys. you can obtain a representative corpus of old chinese sourced from the kanseki repository viadirect-phonology/ect-krp.a simple invocation ofdphonmight look like:$dphontext_a.txttext_b.txtwhich would look for phonetically similar passages betweentext_aandtext_b. the output will be a list of sequences and their phonemic transcriptions, with an identifier based on the file's name and an indicator of where in the text the sequence occurs:1.text_a(2208–2216)：夏后啟曰以為可為故為之為之天下弗能*ləʔɢʷrajkʰˤajʔɢʷrajkˤaʔsɢʷrajtəɢʷrajtə2.text_b(3340–3348)：不可弗爲以爲可故爲之爲之繇其道物*ləʔɢʷrajkʰˤajʔkˤaʔsɢʷrajtəɢʷrajtəpitthe numbers next to the identifiers aretoken indices, and may vary depending on how the text is tokenized –dphoncurrently uses character-based tokenization. whitespace will be removed, and the output will be aligned to make it easier to spot differences between the two sequences. by default, insertions are highlighted in green, and mismatches (differences between the two sequences) are highlighted in red. additional (non-matching) context added to either side of match sequences is displayed using a dimmed color (see "advanced usage" below for more information on colorization).matches are sorted by the ratio of their phomenic similarity to their graphic similarity – in other words, matches between texts that sound highly similar but were written very differently will be at the top of the list.by default,dphononly returns matches that display at least one instance ofgraphic variation– a case where two different graphemes are used in the same place to represent the same sound. these cases are highlighted in blue. if you're interested in all instances of reuse, regardless of graphic variation, you can use the--allflag:$dphon--alltext_a.txttext_b.txtyou can view the full list of command options with:$dphon--helpthis tool is under active development, and results may vary. to find the version you are running:$dphon--versionadvanced usageby default,dphonuses your system's$PAGERto display output, since the results can be quite long. on MacOS and Linux, this will likely beless, which supports additional options like searching through the output once it's displayed. for more information, see the man page:$manlessdphoncan colorize output for nicer display in the terminal if your pager supports it. to enable this behavior on MacOS and Linux, setLESS=R:$exportLESS=Rif you want to save the results of the run to a file, you can use redirection. this is useful when writing structured formats like .csv and .jsonl. you can also write html to preserve colors:$dphon-ohtmlfiles/*.txt>results.htmlalternatively, you can pipe the output ofdphonto another utility likesedfor filtering the results further. for example, you could strip out the ideographic spacefrom results to remove the alignments:$dphonfiles/*.txt|sed's/　//g'methodologymatching sequences are determined by a "dictionary" file that represents a particular reconstruction of old chinese phonology. these data structures perform grapheme-to-phoneme conversion, yielding the associated sound for each character:"埃": "qˤə"
"哀": "ʔˤəj"
"藹": "qˤats"
...if two characters have the same phonemes, they're treated as a match. for characters with multiple readings,dphoncurrently chooses the first available reading for comparison. more work is planned for version 3.0 to address this shortcoming.in version 1.0,dphon's default reconstruction was based on Schuessler 20071, but used a single "dummy" character to represent all the lexemes in a rhyming group.the dictionarywas compiled by John O'Leary (@valgrinderror) and Gian Duri Rominger (@GDRom). since version 2.0,dphonusesa dictionarybased on the Baxter-Sagart 2014 reconstruction2, with additional work by Rominger.the matching algorithm is based on Paul Vierthaler'schinesetextreuseproject3, with some modifications. it uses aBLAST-like strategy to identify initial match candidates, and then extend them via phoneticedit distancecomparison. finally, the results are aligned using a version of theSmith-Waterman algorithmthat operates on phonemes, powered by thelingpylibrary4.development setupfirst, clone the repository:$gitclonehttps://github.com/direct-phonology/dphon.git
$cddphonthen, to create and activate a virtual environment (recommended):$python-mvenvvenv
$sourcevenv/bin/activateinstall dependencies:$pipinstall-rdev-requirements.txtfinally, install the package itself in development mode:$pipinstall-e.now your changes will be automatically picked up when you rundphon.pull requests can be made againstmain.code documentationcode documentation isavailable on github pagesand is generated withpdoc3.to build the docs:$pdoc--html--output-dirdocsdphontestsunit tests are written withunittest. you can run them with:$python-munittestreleasesthe package is built and published to pyPI automatically usingtwinewhen using GitHub's release functionality.make sure the version number indphon/__init__.pyis correct!1Schuessler, Axel (2007), _ABC Etymological Dictionary of Old Chinese_, Honolulu: University of Hawaii Press, ISBN 978-0-8248-2975-9.2Baxter, William H.; Sagart, Laurent (2014),Old Chinese: A New Reconstruction, Oxford University Press, ISBN 978-0-19-994537-5.3Vierthaler, Paul, and Mees Gelein. “A BLAST-Based, Language-Agnostic Text Reuse Algorithm with a MARKUS Implementation and Sequence Alignment Optimized for Large Chinese Corpora,” April 26, 2019.https://doi.org/10.31235/osf.io/7xpqe.4List, Johann-Mattis; Greenhill, Simon; Tresoldi, Tiago; and Forkel, Robert (2019):LingPy. A Python library for historical linguistics. Version 2.6.5. URL:http://lingpy.org, DOI:https://zenodo.org/badge/latestdoi/5137/lingpy/lingpy. With contributions by Christoph Rzymski, Gereon Kaiping, Steven Moran, Peter Bouda, Johannes Dellert, Taraka Rama, Frank Nagel. Jena: Max Planck Institute for the Science of Human History.