Hands-on QuEst++
Carolina Scarton, Gustavo Paetzold and Lucia Specia
University of Sheffield

https://github.com/ghpaetzold/questplusplus

COLING, Osaka, 11 Dec 2016

Definition
Goal: framework to explore features for QE

Definition
Goal: framework to explore features for QE
I Feature extractors: for 150+ features of all types: Java

Definition
Goal: framework to explore features for QE
I Feature extractors: for 150+ features of all types: JavaI Machine learning: wrappers for a number of algorithms inthe scikit-learn toolkit, grid search, feature selection: PythonDefinition
Goal: framework to explore features for QE
I Feature extractors: for 150+ features of all types: JavaI Machine learning: wrappers for a number of algorithms inthe scikit-learn toolkit, grid search, feature selection: Python

Open source: http://www.quest.dcs.shef.ac.uk/

Definition

New release: word and document-level feature extraction andmachine learning added

Definition

New release: word and document-level feature extraction andmachine learning added
I Feature extractors: 40 features for word-level and 79

features for document-levelDefinition

New release: word and document-level feature extraction andmachine learning added
I Feature extractors: 40 features for word-level and 79

features for document-level
I Machine learning: support to Conditional Random Fields(CRF) added for word-level models

Definition

New release: word and document-level feature extraction andmachine learning added
I Feature extractors: 40 features for word-level and 79

features for document-level
I Machine learning: support to Conditional Random Fields(CRF) added for word-level models
I Another important improvement: changes on the corefunctionalities

System and baseline features required

I Java 8 (OpenJDK or Oracle versions)System and baseline features required

I Java 8 (OpenJDK or Oracle versions)
I Sentence and word-level baseline features

Perl 5 (or above)
SRILM
I Tokenizer and Truecaser (from Moses toolkit)
IISystem and baseline features required

I Java 8 (OpenJDK or Oracle versions)
I Sentence and word-level baseline features

Perl 5 (or above)
SRILM
I Tokenizer and Truecaser (from Moses toolkit)
II

I Word-level features

Stanford Core NLP 3.5.1 models
Stanford Core NLP Spanish model
I Universal WordNet plugin
IIBasic Usage - Sentence-leveljava -cp QuEst++.jar shef.mt.SentenceLevelFeatureExtractor-tok -case true-lang <<lang source>> <<lang target>>-input <<input source>> <<input target>>-config <<config file>>

Input files

I Word and sentence levels: file with one sentence per lineInput files

I Word and sentence levels: file with one sentence per line
I Document level: file with paths for documents

Input files

I Word and sentence levels: file with one sentence per line
I Document level: file with paths for documents

Files from source and target should have the same numberof lines

Folders

I src: source code
I lang resources: folder containing all language resources

required for the features
I lib: external libraries needed for feature extraction
I config: configuration files for running QuEst++
I input: auxiliary input folder
I output: output folder

Adding a new feature

I Example with sentence-level feature extractor

Adding a new feature

I Example with sentence-level feature extractor
I New feature: complex words per sentence (averaged by thelength of sentence)Adding a new feature

I Example with sentence-level feature extractor
I New feature: complex words per sentence (averaged by thelength of sentence)
I Language Resource: list of simple words (LSW)

Adding a new feature

I Example with sentence-level feature extractor
I New feature: complex words per sentence (averaged by thelength of sentence)
I Language Resource: list of simple words (LSW)
I Idea: count words not in the LSW and normalise by numberof words in the sentenceAdding a new feature

I Creating a processor for the new feature
IPackage: shef.mt.tools

Adding a new feature

I Creating a processor for the new feature
II

Package: shef.mt.tools
Function: prepare resources to be used by features

Adding a new feature

I Creating a processor for the new feature

Package: shef.mt.tools
Function: prepare resources to be used by features
I Extends ResourceProcessor class: add the resources to thesentence (processNextSentence method)IIAdding a new feature

I Creating a processor for the new feature

Package: shef.mt.tools
Function: prepare resources to be used by features
I Extends ResourceProcessor class: add the resources to thesentence (processNextSentence method)
I It is useful because a unique processor can be used by severalfeatures

IIAdding a new feature

I Create a new Java class called

ComplexWordsProcessor.java
I

Package: shef.mt.tools

Adding a new feature

I Create a new Java class called

ComplexWordsProcessor.java
I
I

Package: shef.mt.tools
Extends: ResourceProcessor classAdding a new feature

I Create a new Java class called

ComplexWordsProcessor.java
Package: shef.mt.tools
Extends: ResourceProcessor class
I Read the LSW and store it on a ArrayListII

Adding a new feature

I Creating a class for the new feature
I

Package: shef.mt.features.impl.bb

Adding a new feature

I Creating a class for the new feature
II

Package: shef.mt.features.impl.bb
Extends Feature class: run method - feature extraction itself

Adding a new feature

I Creating a class for the new feature

Package: shef.mt.features.impl.bb
Extends Feature class: run method - feature extraction itself
I Feature classes are usually named following a number order(e.g. Feature1001, Feature1002)

IIAdding a new feature

I Create a new Java class called Feature7001.java
I

Package: shef.mt.features.impl.bb

Adding a new feature

I Create a new Java class called Feature7001.java
IIPackage: shef.mt.features.impl.bb
Extends: Feature classAdding a new feature

I Create a new Java class called Feature7001.java

Package: shef.mt.features.impl.bb
Extends: Feature class
I Get the ArrayList from the ComplexWordsProcessor classand calculate the featureIIAdding a new feature

I Create a new Java class called Feature7001.java

Package: shef.mt.features.impl.bb
Extends: Feature class
I Get the ArrayList from the ComplexWordsProcessor classand calculate the feature
I Also define the resource that will be required for this featureIIAdding a new feature

I Feature configuration file
IFolder: config/features

Adding a new feature

I Feature configuration fileII

Folder: config/features
XML file with the featureset that will be executed

Adding a new feature

I Feature configuration file
I

Create a file named features complex words.xml inside thefolder config/features

Adding a new feature

I Feature configuration file

Create a file named features complex words.xml inside thefolder config/features
I Add the new feature to this fileIAdding a new feature

I Configuration fileI

Folder: config

Adding a new feature

I Configuration fileII

Folder: config
For sentence-level: config.sentence-level.properties

Adding a new feature

I Configuration file

Folder: config
For sentence-level: config.sentence-level.properties
I Contains basic configuration for the system and paths toresources and toolsIIAdding a new feature

I Configuration fileI

Add the resource source.simplewords to the configuration fileAdding a new feature

I Configuration fileII

Add the resource source.simplewords to the configuration file
Change the option featureConfig to the path tofeatures complex words.xml

Adding a new feature

I SentenceLevelProcessorFactory.java
I

Package: shef.mt.tools

Adding a new feature

I SentenceLevelProcessorFactory.java
IIPackage: shef.mt.tools
Function: create all processors required by the features

Adding a new feature

I SentenceLevelProcessorFactory.java

Package: shef.mt.tools
Function: create all processors required by the features
I Only generate processors that will be used (improvement ofQuEst++)II

Adding a new feature

I SentenceLevelProcessorFactory.java

Package: shef.mt.tools
Function: create all processors required by the features
I Only generate processors that will be used (improvement ofQuEst++)I It is the connection between features and configuration fileII

Adding a new feature

I SentenceLevelProcessorFactory.java
I

Package: shef.mt.tools

Adding a new feature

I SentenceLevelProcessorFactory.java
I
I

Package: shef.mt.tools
Add an if block containing the calling to a method called
getComplexWordsProcessor

Adding a new feature

I SentenceLevelProcessorFactory.java

Package: shef.mt.tools
Add an if block containing the calling to a method called
getComplexWordsProcessor
I Implement getComplexWordsProcessor methodIIBuild

I NetBeans 8.1
I ant “-Dplatforms.JDK 1.8.home=/usr/lib/jvm/java-8-<<version>>”

Run

java -cp QuEst++.jar
shef.mt.SentenceLevelFeatureExtractor-tok -case true-lang <<lang source>> <<lang target>>-input <<input source>> <<input target>>-config <<config file>>
Check the file output.txt inside output/test

System requirements

I Python 2.7.6 (or above - only 2.7 stable distributions)System requirements

I Python 2.7.6 (or above - only 2.7 stable distributions)
I SciPy and NumPy (SciPy >=0.9 and NumPy >=1.6.1)
I scikit-learn (version 0.15.2)
I PyYAMLSystem requirements

I Python 2.7.6 (or above - only 2.7 stable distributions)
I SciPy and NumPy (SciPy >=0.9 and NumPy >=1.6.1)
I scikit-learn (version 0.15.2)
I PyYAML
I GPy
I CRFsuiteFolders

I learning: main folder
I config: configuration files
I src: source code files
I data: example data (same format as output of featureextractor) + scores

Runpython src/learn model.py config/<<config file>>

Machine learning algorithms

I SVR
I SVC
I LassoCV
I LassorLars
I LassorLarsCV
I GP (implemented using GPy - need some code update)
I CRF (implemented using CRFsuite)Adding a machine learning algorithm

Exemple using an algorithm from scikit-learn

Adding a machine learning algorithm

Exemple using an algorithm from scikit-learn
I Algorithm: Ridge: Linear least squares with l2 regularization

Adding a machine learning algorithm

Exemple using an algorithm from scikit-learn
I Algorithm: Ridge: Linear least squares with l2 regularization
I Package: sklearn.linear.model.Ridge

Adding a machine learning algorithm

Exemple using an algorithm from scikit-learn
I Algorithm: Ridge: Linear least squares with l2 regularization
I Package: sklearn.linear.model.Ridge
I Idea: include the algorithm on the available codeAdding a machine learning algorithm

learn model.py
I Main class of QuEst++ machine learning module

Adding a machine learning algorithm

learn model.py
I Main class of QuEst++ machine learning module
I Method: set learning method(config, X train, y train)

Adding a machine learning algorithm

learn model.py
I Main class of QuEst++ machine learning module
I Method: set learning method(config, X train, y train)
I Create estimators for the new algorithm

Configuration file

I Folder: config

Configuration file

I Folder: config
I Files follows the YAML format

Configuration file

I Folder: config
I Files follows the YAML format
I Open the file svr.cfg to see an example

Configuration file

I Folder: configI Files follows the YAML format
I Open the file svr.cfg to see an example

Create a new file called ridge.cfg and follow the structured
YAML to provide parameters for the modelRunpython src/learn model.py config/ridge.cfg

Hands-on QuEst++
Carolina Scarton, Gustavo Paetzold and Lucia Specia
University of Sheffield

https://github.com/ghpaetzold/questplusplus

COLING, Osaka, 11 Dec 2016