Package: disco-dop
The aim of this project is to parse discontinuous constituents in natural
language using Data-Oriented Parsing (DOP), with a focus on global world
domination. The grammar is extracted from a treebank of sentences annotated
with (discontinuous) phrase-structure trees. Concretely, this project provides
a statistical constituency parser with support for discontinuous constituents
and Data-Oriented Parsing. Discontinuous constituents are supported through the
grammar formalism Linear Context-Free Rewriting System (LCFRS), which is a
generalization of Probabilistic Context-Free Grammar (PCFG). Data-Oriented
Parsing allows re-use of arbitrary-sized fragments from previously seen
sentences using Tree-Substitution Grammar (TSG).Contents of this README:FeaturesInstallationDebian, Ubuntu based systemsOther Linux systemsMac OS XOther systemsUsage, documentationGrammarsAcknowledgmentsReferencesFeaturesGeneral statistical parsing:grammar formalisms: PCFG, PLCFRSextract treebank grammar: trees decomposed into productions, relative
frequencies as probabilitiesexactk-best list of derivationscoarse-to-fine pruning: posterior threshold,k-best coarse-to-fineDOP specific (parsing with tree fragments):implementations: Goodman’s DOP reduction, Double-DOP, DOP1.estimators: relative frequency estimate (RFE), equal weights estimate (EWE).objective functions: most probable parse (MPP),
most probable derivation (MPD), most probable shortest derivation (MPSD),
most likely tree with shortest derivation (SL-DOP).marginalization: n-best derivations, sampled derivations.InstallationRequirements:Python 3.3+http://www.python.org(headers required, e.g. python3-dev package)Cython 0.21+http://www.cython.orgNumpy 1.5+http://numpy.org/Python 2.7 is supported, but Python 3 is recommended.
Install thefuturespackage when running Python 2.7.Debian, Ubuntu based systemsTo compile the latest development version on anUbuntusystem,
run the following sequence of commands:sudo apt-get install build-essential python3-dev python3-numpy python3-pip git
git clone --depth 1 git://github.com/andreasvc/disco-dop.git
cd disco-dop
pip3 install --user -r requirements.txt
make installThe--useroption means the packages will be installed to your home
directory which does not require root privileges. Make sure that~/.local/bindirectory is in your PATH, or add it as follows:echo export PATH=$HOME/.local/bin:$PATH >> ~/.bashrcOther Linux systemsThis assumes no root access, but assumes thatgccis installed.Set environment variables so that software can be installed to the home directory
(replace with equivalent for your shell if you do not use bash):mkdir -p ~/.local
echo export PATH=$HOME/.local/bin:$PATH >> ~/.bashrc
echo export LD_LIBRARY_PATH=$HOME/.local/lib:/usr/lib64:/usr/lib >> ~/.bashrc
echo export PYTHONIOENCODING="utf-8" >> ~/.bashrcAfter this, re-login or restart the shell to activate these settings.
Install Python 3 from source, if not installed already.
Python may require some libraries such aszlibandreadline;
installation steps are similar to the ones below:wget http://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz
tar -xzf Python-*.tgz
cd Python-*
./configure --prefix=$HOME/.local --enable-shared
make install && cd ..Check by runningpython3that version 3.5.1 was installed successfully and
is the default.Install the latest development version of discodop:wget https://github.com/andreasvc/disco-dop/archive/master.zip
unzip disco-dop-master.zip
cd disco-dop-master
pip install --user -r requirements.txt
make installMac OS XInstallXcodeandHomebrewInstall dependencies using Homebrew:brew install gcc python3 git
git clone --depth 1 git://github.com/andreasvc/disco-dop.git
cd disco-dop
sudo pip3 install -r requirements.txt
env CC=gcc sudo python setup.py install
sudo makeOther systemsIf you do not run Linux, it is possible to run the code inside a virtual machine.
To do that, installVirtualboxand download the virtual machine imagine with disco-dop pre-installed:http://lang.science.uva.nl/VMs/discodop-vboximage.zipUsage, documentationdiscodop can be used in three ways:through the command line; cf. the manual pages for thediscodopcommand
installed as part of the installation:man discodop.as a library, cf. theAPI referenceandexample notebooksWeb interfacesNB: avoid running discodop from within the source tree, to ensure that the
installed versions of modules are imported.The documentation can be found athttp://discodop.readthedocs.ioGrammarsCf.https://lang.science.uva.nl/grammars/The English, German, and Dutch grammars are described invan Cranenburgh et al., (2016);
the French grammar appears inSangati & van Cranenburgh (2015).
For comparison, there is also an English grammar without discontinuous
constituents (ptb-nodisc).AcknowledgmentsThe Tree data structures intree.pyand the simple binarization algorithm
intreetransforms.pywere taken fromNLTK.
The Zhang-Shasha tree-edit distance algorithm intreedist.pywas taken fromhttps://github.com/timtadh/zhang-shashaElements of the PLCFRS parser and punctuation re-attachment are based on code
fromrparse. Various other bits inspired
by the Stanford parser, Berkeley parser, Bubs parser, &c.ReferencesPlease cite the following paper if you use this code in the context of a
publication:@article{vancranenburgh2016disc,
    title={Data-Oriented Parsing with discontinuous constituents and function tags},
    author={van Cranenburgh, Andreas and Remko Scha and Rens Bod},
    journal={Journal of Language Modelling},
    year={2016},
    volume={4},
    number={1},
    pages={57--111},
    url={http://dx.doi.org/10.15398/jlm.v4i1.100}
}