Package: genie-parser
Genie-parserA neural-network based semantic parser, designed to be used in conjuction withgenie-toolkit,
a set of tools to generate large scale semantic parsing datasets quickly.Genie was described in the paper:Genie: A Generator of Natural Language Parsers for Compositional Virtual AssistantsGiovanni Campagna (*), Silei Xu (*), Mehrad Moradshahi, and Monica S. LamTo appear inProceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation(PLDI 2019), Phoenix, AZ, June 2019.Genie-parser is part of Almond, a research project of the Mobile and Social
Computing Lab at Stanford University.  You can find more
information athttp://almond.stanford.edu/.Installationgenie-parser depends on numpy, Tensorflow 1.12 for Python 3, and a number of
other python modules. To install the dependencies, use:pip3 install -r requirements.txtIt's recommended to install numpy from distribution packages, not
pip because it's faster and more reliable.You must also installtensor2tensorusing our own fork, using the version
indicated in requirements.txt.
Do not install tensor2tensor from pypi, as that is not compatible.Please see the Tensorflow documentation if you wish to use GPU
acceleration. You'll need to install nvidia+cuda or amdgpu-pro+rocm
drivers, and then install "tensorflow-gpu" from pip, or build
Tensorflow from source.genie-parser has been tested successfully on Fedora 25 to 28
x86_64 with CPU and Nvidia GPU acceleration, as well as Ubuntu 16.04,
18.04. On RHEL/CentOS 7, you will need to use python 3.6 from therh-python36software collection. On Fedora 29 and later, the default is python 3.7, which is not
compatible with Tensorflow, so you will need to install python 3.6 separately.It is also possible to use thepipenvtool to set up a virtualenv and install the dependencies inside it.TrainingTo train a new model, you should do the follow:Unpack the dataset (*.tsv files) into adataset/directory.Download the word embeddings. We recommend using Glove 42B 300-dimensional,
which can be downloaded fromhttps://nlp.stanford.edu/data/glove.42B.300d.zip,
or from our mirror athttps://oval.cs.stanford.edu/data/glove/glove.42B.300d.zip.
Set theGLOVEenvironment variable to the path of uncompressed text file.You can skip this step, in which case, theluinet-datagenscript will download
the recommended GloVe file automatically.Prepare the working directory:luinet-datagen --src_data_dir ./dataset --data_dir ./workdir --thingpedia_snapshot [SNAPSHOT]
   --problem semparse_thingtalk_noquoteThis script computes the input dictionary, downloads a snapshot of Thingpedia,
and computes a subset of the word embedding matrix corresponding to the dictionary.Use the optional SNAPSHOT argument to choose which Thingpedia snapshot to train
against, or pass -1 for the latest content of Thingpedia. Be aware that
using -1 might make the results impossible to reproduce.
This script will verify that the dataset is compatible with the
Thingpedia snapshot.The--problemparameter should be used to chosen to match the target language.
The meaning is similar to that in thetensor2tensorlibrary. Seegenieparser/tasks/init.pyfor a
list of available problems.Train:genie-trainer --data_dir ./workdir --output_dir ./workdir/model
  --model genie_copy_seq2seq
  --hparams_set 'lstm_genie'
  --hparams_overrides ''
  --decode_hparams 'beam_size=20,return_beams=true'
  --problem 'semparse_thingtalk_noquote'
  --eval_early_stopping_metric 'metrics-semparse_thingtalk_noquote/accuracy'
  --noeval_early_stopping_metric_minimizeSeeluinet-trainer --helpfor a description of each options. Available hparams sets are
ingenieparser/layers/hparams.py, and available models
are atgenieparser/models/init.py.During training, you can usetensorboardto visualize
progress:tensorboard --logdir ./workdir/modelAfter training, you can extract the metrics of the best model on the validation set with:genie-print-metrics --output_dir ./workdir/model
  --eval_early_stopping_metric 'metrics-semparse_thingtalk_noquote/accuracy'
  --noeval_early_stopping_metric_minimizeEvaluate:genie-trainer --data_dir ./workdir --output_dir ./workdir/model
  --model genie_copy_seq2seq
  --hparams_set 'lstm_genie'
  --hparams_overrides ''
  --decode_hparams 'beam_size=20,return_beams=true'
  --problem 'semparse_thingtalk_noquote'
  --eval_early_stopping_metric 'metrics-semparse_thingtalk_noquote/accuracy'
  --noeval_early_stopping_metric_minimize
  --schedule evaluateAdd--eval_use_test_setto use the test set instead of the validation set.You can also evaluate on a specific saved model (such as the best model according to the metric
on the validation set) using the flag--checkpoint_path ./workdir/model/export/best/.../variables/variablesTo deploy the model, point the server to a saved model directory fromworkdir/model/export/best,
by writing a configuration fileserver.confcontaining:[models]
en=<path-to-saved-model>Run the server with:genie-server --config-file <path-to-server.conf>By default, the server runs at port 8400. You can change that by editing the server.conf file.
An the example server.conf file is provided in thedatafolder, which describes all available
options, including SSL and privilege separation.The server expects to connect to a TokenizerService (provided byAlmond Tokenizer) on
localhost, port 8888.