Package: pyfn
pyfnWelcome topyfn, a Python module to process FrameNet annotation.pyfncan be used to:convertdata to and from FRAMENET XML, SEMEVAL XML, SEMAFOR CoNLL, BIOS and CoNLL-XpreprocessFrameNet data using a standardized state-of-the-art pipelinerunthe SEMAFOR,  OPEN-SESAME and SIMPLEFRAMEID frame semantic parsers for frame and/or argument identification on the FrameNet 1.5, 1.6 and 1.7 datasetsbuildyour own frame semantic parser using a standard set of python models to marshall/unmarshall FrameNet XML dataThis repository also accompanies the (Kabbach et al., 2018) paper:@InProceedings{C18-1267,
  author = 	"Kabbach, Alexandre
		and Ribeyre, Corentin
		and Herbelot, Aur{\'e}lie",
  title = 	"Butterfly Effects in Frame Semantic Parsing: impact of data processing on model ranking",
  booktitle = 	"Proceedings of the 27th International Conference on Computational Linguistics",
  year = 	"2018",
  publisher = 	"Association for Computational Linguistics",
  pages = 	"3158--3169",
  location = 	"Santa Fe, New Mexico, USA",
  url = 	"http://aclweb.org/anthology/C18-1267"}DependenciesOn Unix, you may need to install the following packages:libxml2 libxml2-dev libxslt1-dev python-3.x-devInstallpip3 install pyfnUseWhen usingpyfn, your FrameNet splits directory structure should follow:.
|-- fndata-1.x-with-dev
|   |-- train
|   |   |-- fulltext
|   |   |-- lu
|   |-- dev
|   |   |-- fulltext
|   |   |-- lu
|   |-- test
|   |   |-- fulltext
|   |   |-- lu
|   |-- frame
|   |-- frRelation.xml
|   |-- semTypes.xmlConversionpyfncan be used to convert data to and from:FRAMENET XML: the format of the released FrameNet XML dataSEMEVAL XML: the format of the SEMEVAL 2007 shared task 19 on frame semantic structure extractionSEMAFOR CoNLL: the format used by the SEMAFOR parserBIOS: the format used by the OPEN-SESAME parserCoNLL-X: the format used by various state-of-the-art POS taggers and dependency
parsers (see preprocessing considerations for frame semantic parsingbelow)As well as to generate the.csvhierarchy files used by both SEMAFOR and
OPEN-SESAME parsers to integrate the hierarchy feature (see (Kshirsagar et al., 2015) for details).For an exhaustive description of all formats, check outFORMAT.md.HowToThe following sections provide examples of commands to convert FN data
to and from different formats. All commands can make use of the following options:--splits: specify which splits should be converted.--splits trainwill generate all
train/dev/test splits, according to data found under the fndata-1.x/{train/dev/test}
directories.--splits devwill generate the dev and test splits according to data found under
the fndata-1.x/{dev/test} directories. This option will skip the train splits but generate the
same dev/test splits that would have been generated with--splits train.--splits testwill
generate the test splits according to data found under the fndata-1.x/test directory, and skip
the train/dev splits. The test splits generated with--splits testwill be the same as those
generated with the--splits trainand--splits dev. Default to--splits test.--output_sentences: if specified, will output a.sentencesfile
in the process, containing all raw annotated sentences, one sentence per line.--with_exemplars: if specified, will process the exemplars (data under
theludirectory) in addition to fulltext.--filter: specify data filtering options (see details below).For details onpyfnusage, do:pyfn--help
pyfngenerate--help
pyfnconvert--helpFrom FN XML to BIOSTo convert data from FrameNet XML format to BIOS format, do:pyfnconvert\--fromfnxml\--tobios\--source/abs/path/to/fndata-1.x\--target/abs/path/to/xp/data/output/dir\--splitstrain\--output_sentences\--filteroverlap_fesUsing--filter overlap_feswill skip all annotationsets with overlapping
frame elements, as those cases are not supported by the BIOS format.From FN XML to SEMAFOR CoNLLTo generate thetrain.frame.elementsfile used to train SEMAFOR, and the{dev,test}.framesfile used for decoding, do:pyfnconvert\--fromfnxml\--tosemafor\--source/abs/path/to/fndata-1.x\--target/abs/path/to/xp/data/output/dir\--splitstrain\--output_sentencesFrom FN XML to SEMEVAL XMLTo generate the{dev,test}.gold.xmlgold files in SEMEVAL format for scoring, do:pyfnconvert\--fromfnxml\--tosemeval\--source/abs/path/to/fndata-1.x\--target/abs/path/to/xp/data/output/dir\--splits{dev,test}From BIOS to SEMEVAL XMLTo convert the decoded BIOS files{dev,test}.bios.semeval.decodedof
OPEN-SESAME to SEMEVAL XML format for scoring, do:pyfnconvert\--frombios\--tosemeval\--source/abs/path/to/{dev,test}.bios.semeval.decoded\--target/abs/path/to/output/{dev,test}.predicted.xml\--sent/abs/path/to/{dev,test}.sentencesFrom SEMAFOR CoNLL to SEMEVAL XMLTo convert the decoded{dev,test}.frame.elementsfiles of SEMAFOR to
SEMEVAL XML format for scoring, do:pyfnconvert\--fromsemafor\--tosemeval\--source/abs/path/to/{dev,test}.frame.elements\--target/abs/path/to/output/{dev,test}.predicted.xml\--sent/abs/path/to/{dev,test}.sentencesGenerate the hierarchy.csvfilespyfngenerate\--source/abs/path/to/fndata-1.x\--target/abs/path/to/xp/data/output/dirTo also process exemplars, add the--with_exemplarsoptionPreprocessing and Frame Semantic Parsingpyfnships in with a set of bash scripts to preprocess FrameNet data with
various POS taggers and dependency parsers, as well as to perform frame
semantic parsing with a variety of open-source parsers.Currently supported POS taggers include:MXPOST (Ratnaparkhi, 1996)NLP4J (Choi, 2016)Currently supported dependency parsers include:MST (McDonald et al., 2006)BIST BARCH (Kiperwasser and Goldberg, 2016)BIST BMST (Kiperwasser and Goldberg, 2016)Currently supported frame semantic parsers include:SIMPLEFRAMEID (Hartmann et al., 2017) for frame identificationSEMAFOR (Kshirsagar et al., 2015) for argument identificationOPEN-SESAME (Swayamdipta et al., 2017) for argument identificationTo request support for a POS tagger, a dependency parser or a frame semantic
parser, please create anissueon Github/Gitlab.DownloadTo run the preprocessing and frame semantic parsing scripts, first download:data.7zcontaining all the FrameNet splits for FN 1.5 and FN 1.7wgetbackup.3azouz.net/pyfn/data.7zlib.7zcontaining all the different external softwares (taggers, parsers, etc.)wgetbackup.3azouz.net/pyfn/lib.7zresources.7zcontaining all the required resourceswgetbackup.3azouz.net/pyfn/resources.7zscripts.7zcontaining the set of bash scripts to call the different parsers and preprocessing toolkitswgetbackup.3azouz.net/pyfn/scripts.7zExtract the content of all the archives under a
directory namedpyfn. Your pyfn folder structure should look like:.
|-- pyfn
|   |-- data
|   |   |-- fndata-1.5-with-dev
|   |   |-- fndata-1.7-with-dev
|   |-- lib
|   |   |-- bistparser
|   |   |-- jmx
|   |   |-- mstparser
|   |   |-- nlp4j
|   |   |-- open-sesame
|   |   |-- semafor
|   |   |-- semeval
|   |-- resources
|   |   |-- bestarchybrid.model
|   |   |-- bestarchybrid.params
|   |   |-- bestfirstorder.model
|   |   |-- bestfirstorder.params
|   |   |-- config-decode-pos.xml
|   |   |-- nlp4j.plemma.model.all.xz
|   |   |-- sskip.100.vectors
|   |   |-- wsj.model
|   |-- scripts
|   |   |-- CoNLLizer.py
|   |   |-- deparse.sh
|   |   |-- flatten.sh
|   |   |-- ...Please strictly follow this directory structure to avoid unexpected errors.pyfnrelies on a lot of relative path resolutions to make scripts calls shorter, and changing this directory structure can break everythingSetup NLP4J for POS taggingTo use NLP4J for POS tagging, modify theresources/config-decode-pos.xmlfile by replacing the models.pos absolute path to
yourresources/nlp4j.plemma.model.all.xz:<configuration>...<models><pos>/absolute/path/to/pyfn/resources/nlp4j.plemma.model.all.xz</pos></models></configuration>Setup DyNET for BIST or OPEN-SESAMEIf you intend to use the BIST parser for dependency parsing or
OPEN-SESAME for frame semantic parsing, you will need
to install DyNET 2.0.2 via:pip install dynet=2.0.2If you experience problems installing DyNET via pip, follow:https://dynet.readthedocs.io/en/2.0.2/python.htmlSetup SEMAFORTo use the SEMAFOR frame semantic parser, modify thescripts/setup.shfile:# SEMAFOR options to be changed according to your envexportJAVA_HOME_BIN="/abs/path/to/java/jdk/bin"exportnum_threads=2# number of threads to useexportmin_ram=4g# min RAM allocated to the JVM in GB. Corresponds to the -Xms argumentexportmax_ram=8g# max RAM allocated to the JVM in GB. Corresponds to the -Xmx argument# SEMAFOR hyperparametersexportkbest=1# keep k-best parseexportlambda=0.000001# hyperparameter for argument identification. Refer to Kshirsagar et al. (2015) for details.exportbatch_size=4000# number of batches processed at once for argument identification.exportsave_every_k_batches=400# for argument identificationexportnum_models_to_save=60# for argument identificationSetup SIMPLEFRAMEIDIf you intend to use SIMPLEFRAMEID for frame identification, you will need to install the following packages (on python 2.7):pip install keras==2.0.6 lightfm==1.13 sklearn numpy==1.13.1 networkx==1.11 tensorflow==1.3.0Using the SEMEVAL PERL evaluation scriptsIf you intend to use the SEMEVAL perl evaluation scripts, make sure
to have theApp::cpanminusandXML::Parsermodules installed:cpan App::cpanminus
cpanm XML::ParserUsing bash scriptsEach script comes with a helper: check it out with--help!Careful!most scripts expect data output bypyfn convert ...to be located underpyfn/experiments/xp_XYZ/datawhereXYZstands for
the experiments number and is specified using the-x XYZargument, and where
theexperimentsdirectory is located at the same level as thescriptsdirectory. This opinionated choice has proven extremely useful in launching
scripts by batch on a large set of experiments as it avoids having to input
the full path each time.Make sure to usepyfnconvert\--from...\--to...\--source...\--target/abs/path/to/pyfn/experiments/xp_XYZ/data\--splits...BEFOREcallingpreprocess.sh,prepare.sh,semafor.shoropen-sesame.shpreprocess.shUsepreprocess.shto POS-tag and dependency-parse FrameNet splits generated
withpyfn convert .... The helper should display:Usage: ${0##*/} [-h] -x XP_NUM -t {mxpost,nlp4j} -p {semafor,open-sesame} [-d {mst,bmst,barch}] [-v]
Preprocess FrameNet train/dev/test splits.

  -h, --help                           display this help and exit
  -x, --xp      XP_NUM                 xp number written as 3 digits (e.g. 001)
  -t, --tagger  {mxpost,nlp4j}         pos tagger to be used: 'mxpost' or 'nlp4j'
  -p, --parser  {semafor,open-sesame}  frame semantic parser to be used: 'semafor' or 'open-sesame'
  -d, --dep     {mst,bmst,barch}       dependency parser to be used: 'mst', 'bmst' or 'barch'
  -v, --dev                            if set, script will also preprocess dev splitsSuppose you generated FrameNet splits for SEMAFOR using:pyfnconvert\--fromfnxml\--tosemafor\--source/path/to/fndata-1.7-with-dev\--target/path/to/experiments/xp_001/data\--splitstrain\--output_sentencesYou can preprocess those splits with NLP4J and BMST using./preprocess.sh-x001-tnlp4j-dbmst-psemaforprepare.shUseprepare.shto automatically generate misc. data required by the
frame semantic parsing pipeline, such as gold SEMEVAL XML files for scoring,
theframenet.frame.element.mapand the hierarchy.csvfiles
used by SEMAFOR, or theframes.xmlandfrRelations.xmlfiles used by
both SEMAFOR and OPEN-SESAME. The helper should display:Usage: ${0##*/} [-h] -x XP_NUM -p {semafor,open-sesame} -s {dev,test} -f FN_DATA_DIR [-u] [-e]
Prepare misc. data for frame semantic parsing.

  -h, --help                                   display this help and exit
  -x, --xp              XP_NUM                 xp number written as 3 digits (e.g. 001)
  -p, --parser          {semafor,open-sesame}  frame semantic parser to be used: 'semafor' or 'open-sesame'
  -s, --splits          {dev,test}             which splits to score: dev or test
  -f, --fn              FN_DATA_DIR            absolute path to FrameNet data directory
  -u, --with_hierarchy                         if specified, will use the hierarchy feature
  -e, --with_exemplars                         if specified, will use the exemplarsSuppose you generated FrameNet splits for SEMAFOR using:pyfnconvert\--fromfnxml\--tosemafor\--source/path/to/fndata-1.7-with-dev\--target/path/to/experiments/xp_001/data\--splitstrain\--output_sentencesYou can prepare SEMAFOR data using:./prepare.sh-x001-psemafor-stest-f/path/to/fndata-1.7-with-devframeid.shUseframeid.shto perform frame identification using SIMPLEFRAMEID.
The helper should display:Usage: ${0##*/} [-h] -m {train,decode} -x XP_NUM [-p {semafor,open-sesame}]
Perform frame identification.

  -h, --help                            display this help and exit
  -m, --mode                            train on all models or decode using a single model
  -x, --xp       XP_NUM                 xp number written as 3 digits (e.g. 001)
  -p, --parser   {semafor,open-sesame}  formalize decoded frames for specified parserSuppose you generated FrameNet splits for SEMAFOR using:pyfnconvert\--fromfnxml\--tosemafor\--source/path/to/fndata-1.7-with-dev\--target/path/to/experiments/xp_101/data\--splitstrain\--output_sentencesAfter preprocessing, you can train the SIMPLEFRAMEID parser using:./frameid.sh-mtrain-x101and decode (before decoding argument identification) using:./frameid.sh-mdecode-x101-psemaforsemafor.shUsesemafor.shto train the SEMAFOR parser or decode the test/dev splits.
The helper should display:Usage: ${0##*/} [-h] -m {train,decode} -x XP_NUM [-s {dev,test}] [-u]
Train or decode with the SEMAFOR parser.

  -h, --help                             display this help and exit
  -m, --mode            {train,decode}   semafor mode to use: train or decode
  -x, --xp              XP_NUM           xp number written as 3 digits (e.g. 001)
  -s, --splits          {dev,test}       which splits to use in decode mode: dev or test
  -u, --with_hierarchy                   if specified, parser will use the hierarchy featureSuppose you generated FrameNet splits for SEMAFOR using:pyfnconvert\--fromfnxml\--tosemafor\--source/path/to/fndata-1.7-with-dev\--target/path/to/experiments/xp_001/data\--splitstrain\--output_sentencesAfter preprocessing and preparation, you can train the SEMAFOR parser using:./semafor.sh-mtrain-x001and decode the test splits using:./semafor.sh-mdecode-x001-stestopen-sesame.shUseopen-sesame.shto train the OPEN-SESMAE parser or decode the test/dev splits.
The helper should display:Usage: ${0##*/} [-h] -m {train,decode} -x XP_NUM [-s {dev,test}] [-d] [-u]
Train or decode with the OPEN-SESAME parser.

  -h, --help                              display this help and exit
  -m, --mode              {train,decode}  open-sesame mode to use: train or decode
  -x, --xp                XP_NUM          xp number written as 3 digits (e.g. 001)
  -s, --splits            {dev,test}      which splits to use in decode mode: dev or test
  -d, --with_dep_parses                   if specified, parser will use dependency parses
  -u, --with_hierarchy                    if specified, parser will use the hierarchy featureSuppose you generated FrameNet splits for OPEN-SESAME using:pyfnconvert\--fromfnxml\--tobios\--source/path/to/fndata-1.7-with-dev\--target/path/to/experiments/xp_002/data\--splitstrain\--output_sentences\--filteroverlap_fesAfter preprocessing and preparation, you can train the SEMAFOR parser using:./open-sesame.sh-mtrain-x002and decode the test splits using:./open-sesame.sh-mdecode-x002-stestscore.shUsescore.shto obtain P/R/F1 scores for frame semantic parsing on
dev/test splits with the SEMEVAL scoring script, using gold of predicted frames.
The helper should display:Usage: ${0##*/} [-h] -x XP_NUM -p {semafor,open-sesame} -s {dev,test} -f {gold,predicted}
Score frame semantic parsing with a modified version of the SEMEVAL scoring script.

  -h, --help                           display this help and exit
  -x, --xp      XP_NUM                 xp number written as 3 digits (e.g. 001)
  -p, --parser  {semafor,open-sesame}  frame semantic parser to be used: 'semafor' or 'open-sesame'
  -s, --splits  {dev,test}             which splits to score: dev or test
  -f, --frames  {gold,predicted}       score with gold or predicted framesNote that scoring is done with an updated version of the SEMEVAL perl script,
in order to obtain more robust scores across setups. For a full account
of the modifications, refer to (Kabbach et al., 2018) and to the perl scripts
located underlib/semeval/.To obtain scores for SEMAFOR using gold frames on test splits, use:./score.sh-xXYZ-psemafor-stest-fgoldTo obtain scores for SEMAFOR using predicted frames on test splits, use:./score.sh-xXYZ-psemafor-stest-fpredictedReplicationTheexperimentsdirectory provides a detailed set of instructions to
replicate all results reported in (Kabbach et al., 2018) on experimental
butterfly effects in frame semantic parsing. Those instructions can be used
to compare the performances of different frame semantic parsers in various
experimental setups.Marshalling and Unmarshalling FrameNet XML datapyfnprovides a set of Python models to process FrameNet XML data.
Those can be used to help you build you own frame semantic parser.The core of thepyfnmodels is theAnnotationSetcorresponding to an
XML<annotationSet>tag. It stores various information
regarding a given set of FrameNet annotation for a given target in a given sentence.
The notable innovations are thelabelstoreand thevalenceunitstore, which
store FrameNet labels (FE/PT/GF) in their original formats, and in custom
formats which may prove useful for frame semantic parsing.Explore the various models under thepyfn.modelsdirectory of thepyfnpackage.Unmarshalling FrameNet XML dataTo convert a list of fulltext.xml files and/or lu.xml files to a generator
overpyfn.AnnotationSetobjects, with no overlap between train/dev/test splits, use:importpyfn.marshalling.unmarshallers.framenetasfn_unmarshallerif__name__=='__main__':splits_dirpath='/abs/path/to/framenet-1.x-with-dev/'splits='train'with_exemplars=Falseannosets_dict=fn_unmarshaller.get_annosets_dict(splits_dirpath,splits,with_exemplars)splits_dirpathshould point at the directory containing train/dev/test
splits directories (see detailed structureabove).get_annosets_dictwill return a string to AnnotationSet generator dict.
It will ensure no overlap between train/dev/test splits.Callingget_annosets_dictonsplits='test'will return a dictionary
with a single'test'key. Callingget_annosets_dictonsplits='dev'will return a dictionary with two keys:'dev'and'test'.
Callingget_annosets_dictonsplits='train'will return a dictionary
with three keys:'train','dev'and'test'.To iterate over the list of AnnotationSet objects of each key, you can
then do:for(splits,annosets)inannosets_dict.items():print('Iterating over annotationsets for splits:{}'.format(splits))forannosetinannosets:print('annoset with #id ={}'.format(annoset._id))Or simply, to iterate over a specific key values (such as train annosets):forannosetinannosets_dict['train']:print('annoset with #id ={}'.format(annoset._id))Note that for performance, annosets is not a list but a generator.Unmarshalling OPEN-SESAME BIOS dataTo convert a.biosfile with its corresponding.sentencesfile to
a generator overpyfn.AnnotationSetobjects, use:importpyfn.marshalling.unmarshallers.biosasbios_unmarshallerif__name__=='__main__':bios_filepath='/abs/path/to/.bios'sent_filepath='/abs/path/to/.sentences'annosets=bios_unmarshaller.unmarshall_annosets(bios_filepath,sent_filepath)forannosetinannosets:print('annoset with #id ={}'.format(annoset._id))Important!the.biosand.sentencesfiles must have been generated
withpyfn convert ... --to bios ...with the--filter overlap_fesparameter.Unmarshalling SEMAFOR CONLL dataTo convert a.frame.elementsfile with its corresponding.sentencesfile to a generator overpyfn.AnnotationSetobjects, use:importpyfn.marshalling.unmarshallers.semaforassemafor_unmarshallerif__name__=='__main__':semafor_filepath='/abs/path/to/.frame.elements'sent_filepath='/abs/path/to/.sentences'annosets=semafor_unmarshaller.unmarshall_annosets(semafor_filepath,sent_filepath)forannosetinannosets:print('annoset with #id ={}'.format(annoset._id))Unmarshalling SEMEVAL XML dataTo convert a SEMEVAL.xmlfile with its corresponding.sentencesfile to a generator overpyfn.AnnotationSetobjects, use:importpyfn.marshalling.unmarshallers.semevalassemeval_unmarshallerif__name__=='__main__':xml_filepath='/abs/path/to/semeval/.xml'annosetss=semeval_unmarshaller.unmarshall_annosets(xml_filepath)By defaultunmarshall_annosetsfor SEMEVAL will return a generator over embedded annotationsets. To iterate over a single annotationset, use:forannosetsinannosetss:forannosetinannosets:print('annoset with #id ={}'.format(annoset._id))To return a 'flat' list of annosets, pass in theflatten=Trueparameter:importpyfn.marshalling.unmarshallers.semevalassemeval_unmarshallerif__name__=='__main__':xml_filepath='/abs/path/to/semeval/.xml'annosets=semeval_unmarshaller.unmarshall_annosets(xml_filepath,flatten=True)forannosetinannosets:print('annoset with #id ={}'.format(annoset._id))Marshalling to OPEN-SESAME BIOSTo convert a dict ofsplitstopyfn.AnnotationSetobjects to OPEN-SESAME-style.bios, refer topyfn.marshalling.marshallers.bios.marshall_annosets_dictMarshalling to SEMAFOR CONLLTo convert a dict ofsplitstopyfn.AnnotationSetobjects to SEMAFOR-style.frame.elements, refer topyfn.marshalling.marshallers.semafor.marshall_annosets_dictMarshalling to SEMEVAL XMLTo convert a list ofpyfn.AnnotationSetobjects to SEMEVAL-style.xml,
refer topyfn.marshalling.marshallers.semeval.marshall_annosetsMarshalling to .csv hierarchyTo convert a list of relations to a.csvfile, refer topyfn.marshalling.marshallers.hierarchy.marshall_relationsCitationIf you usepyfnplease cite:@InProceedings{C18-1267,
  author = 	"Kabbach, Alexandre
		and Ribeyre, Corentin
		and Herbelot, Aur{\'e}lie",
  title = 	"Butterfly Effects in Frame Semantic Parsing: impact of data processing on model ranking",
  booktitle = 	"Proceedings of the 27th International Conference on Computational Linguistics",
  year = 	"2018",
  publisher = 	"Association for Computational Linguistics",
  pages = 	"3158--3169",
  location = 	"Santa Fe, New Mexico, USA",
  url = 	"http://aclweb.org/anthology/C18-1267"}