Gideon : Implemented a first working version of soft constraint Rule matching.
The new Interface RuleMatcher and implementation class RuleMatcherFactory
introduces methods that match rules, depending on whether what is matched is a non-terminal
or a terminal.
There are several implementations added, depending on whether regular expression matching is allowed and
whether soft syntactic constraint matching is allowed.
The test SoftSyntacticConstraintDecodingTest.java tests for a simple test grammar if the
right number of derivations is created, corresponding to alternate substitutions for the labels
of terminal rules.

       modified:   src/joshua/decoder/chart_parser/Chart.java
       modified:   src/joshua/decoder/chart_parser/DotChart.java
       new file:   src/joshua/decoder/chart_parser/RuleMatcher.java
       new file:   src/joshua/decoder/chart_parser/RuleMatcherFactory.java
       modified:   test/joshua/decoder/ArtificialGrammarAndCorpusCreater.java
       modified:   test/joshua/decoder/SoftSyntacticConstraintDecodingTest.java
       modified:   test/joshua/decoder/TestConfigFileCreater.java
Gideon : Extended the SoftSyntacticConstraintDecodingTest so it tests both normal decoding and decoding with soft syntactic constraints.
Adapted the JoshuaConfiguration so that it now contains a property that can be used to switch soft syntactic constraints decoding
on. Made some other minor improvements / clean ups of the code.

       modified:   src/joshua/decoder/JoshuaConfiguration.java
       modified:   src/joshua/decoder/chart_parser/Chart.java
       modified:   src/joshua/decoder/chart_parser/DotChart.java
       modified:   src/joshua/decoder/chart_parser/RuleMatcherFactory.java
       modified:   test/joshua/decoder/FeatureFunctionsTest.java
       modified:   test/joshua/decoder/SoftSyntacticConstraintDecodingTest.java
       modified:   test/joshua/decoder/TestConfigFileCreater.java
Gideon : Added some documentation explaining broadly the idea behind the new classes implementing Soft Syntactic Constraint Decoding
Gideon : Refactored the RuleMatching behavior in DotChart.java,
in reaction to feedback by Matt Post.
The nonterminal matching should only apply to the nonterminal level.
This simplifies what is required from RuleMatcher : it should only be concerned
with NonterminalMatcher.

For this reason I replaced RuleMatcher and RuleMatcherFactory
with a simpler interface and class
NonterminalMatcher and NonterminalMatcherFactory.
At the monent there are only two types of matching: strict matching and
soft syntactic matching (any nonterminal can substitute for any other
nonterminal, with the exceptions of OOV and GOAL which can only substitute
for themselves).
It is conceivable though that we want to implement other more fine grained
Nonterminal matchers, allowing only some types of nonterminals to match with
some other types perhaps?
For this reason I though it is still cleaner to add a interface and factory
method that implements this behavior, rather than making this all part of
the core class DotChart. The latter class is very fundamental in the  the
decoding process and should not be concerned with the behavior of what
types of nonterminals should match each other, I belief. But were to
put what code and how much to make use of strategy is also a matter of
taste I admit.
Gideon : Added some info output and Assertions to DotChart (1)
as a first step to extending the code to implement soft constraints.
These assertions will later be again removed, but now served to
check wether the assumptions about were terminals and were nonterminals
are added are in fact true. For testing with Assert statements to work
in a java pipeline call, we needed also to have junit to be available
in th dependencies, build file and decoder, which is were the changes to
(2,3,4) come in .

  (1)    modified:   bin/joshua-decoder
  (2)    modified:   build.xml
  (3)    modified:   lib/ivy.xml
  (4)    modified:   src/joshua/decoder/chart_parser/DotChart.java
Gideon : Implemented a first working version of soft constraint Rule matching.
The new Interface RuleMatcher and implementation class RuleMatcherFactory
introduces methods that match rules, depending on whether what is matched is a non-terminal
or a terminal.
There are several implementations added, depending on whether regular expression matching is allowed and
whether soft syntactic constraint matching is allowed.
The test SoftSyntacticConstraintDecodingTest.java tests for a simple test grammar if the
right number of derivations is created, corresponding to alternate substitutions for the labels
of terminal rules.

       modified:   src/joshua/decoder/chart_parser/Chart.java
       modified:   src/joshua/decoder/chart_parser/DotChart.java
       new file:   src/joshua/decoder/chart_parser/RuleMatcher.java
       new file:   src/joshua/decoder/chart_parser/RuleMatcherFactory.java
       modified:   test/joshua/decoder/ArtificialGrammarAndCorpusCreater.java
       modified:   test/joshua/decoder/SoftSyntacticConstraintDecodingTest.java
       modified:   test/joshua/decoder/TestConfigFileCreater.java
Gideon : Extended the SoftSyntacticConstraintDecodingTest so it tests both normal decoding and decoding with soft syntactic constraints.
Adapted the JoshuaConfiguration so that it now contains a property that can be used to switch soft syntactic constraints decoding
on. Made some other minor improvements / clean ups of the code.

       modified:   src/joshua/decoder/JoshuaConfiguration.java
       modified:   src/joshua/decoder/chart_parser/Chart.java
       modified:   src/joshua/decoder/chart_parser/DotChart.java
       modified:   src/joshua/decoder/chart_parser/RuleMatcherFactory.java
       modified:   test/joshua/decoder/FeatureFunctionsTest.java
       modified:   test/joshua/decoder/SoftSyntacticConstraintDecodingTest.java
       modified:   test/joshua/decoder/TestConfigFileCreater.java
Gideon : Refactored the RuleMatching behavior in DotChart.java,
in reaction to feedback by Matt Post.
The nonterminal matching should only apply to the nonterminal level.
This simplifies what is required from RuleMatcher : it should only be concerned
with NonterminalMatcher.

For this reason I replaced RuleMatcher and RuleMatcherFactory
with a simpler interface and class
NonterminalMatcher and NonterminalMatcherFactory.
At the monent there are only two types of matching: strict matching and
soft syntactic matching (any nonterminal can substitute for any other
nonterminal, with the exceptions of OOV and GOAL which can only substitute
for themselves).
It is conceivable though that we want to implement other more fine grained
Nonterminal matchers, allowing only some types of nonterminals to match with
some other types perhaps?
For this reason I though it is still cleaner to add a interface and factory
method that implements this behavior, rather than making this all part of
the core class DotChart. The latter class is very fundamental in the  the
decoding process and should not be concerned with the behavior of what
types of nonterminals should match each other, I belief. But were to
put what code and how much to make use of strategy is also a matter of
taste I admit.
Gideon : Added a new feature function LabelSubstitutionFF.java
which is to be used in combination with soft-syntactic decoding.
The feature function fires boolean features for:

1. The match/mismatch of the rule label with the substituted label
2. A particular subsitution of a substituted label for a certain rule label

Adapted the tests FeatureFunctionTest to test if this feature is firing and
SoftSyntacticConstraintDecodingTest to test if this feature yields different
feature activations if soft syntactic constraint decoding is used as compared
to when normal decoding is used.

Added new ckass src/joshua/decoder/ff/RulePropertiesQuerying.java
to put shared static method used by different feature funtions in a central place.

       modified:   src/joshua/decoder/Decoder.java
       modified:   src/joshua/decoder/ff/LabelCombinationFF.java
       new file:   src/joshua/decoder/ff/LabelSubstitutionFF.java
       new file:   src/joshua/decoder/ff/RulePropertiesQuerying.java
       modified:   test/joshua/decoder/FeatureFunctionsTest.java
       modified:   test/joshua/decoder/SoftSyntacticConstraintDecodingTest.java
Gideon : Removed assertion in
 src/joshua/decoder/ff/LabelSubstitutionFF.java
that leads to problems when calling the decoder without including JUnit in the library path.
Gideon : Implemented a new type of labelsubstitutionfeature
for the combination of a rule - as identified by its labels and whether its gaps are monotone
or inverted (rules with only one gap are considered monotone)
and the substituted labels for those gaps.
This is a more sparse, more specific variant of the basic label substitution feature,
which fires for every susbsitution of one used label for one rule (gap) label.

We added ListUtil.java to store the frequently used stringList methods -
which basically produce different types of Strings from lists of Strings or other objects.

	modified:   src/joshua/decoder/ff/LabelSubstitutionFF.java
	new file:   src/joshua/util/ListUtil.java
Gideon : Adapted the tests and supporting code in SoftSyntacticConstraintDecodingTest.java,
ArtificialGrammarAndCorpusCreater.java and TestConfigFileCreater.java
to produce a grammar variant that also contains an inverted rule for one of the tests, in order
to be able to see that the new feature in LabelSubstitutionFF corresponding to
the combination of the Rule information (Labels + Inverted/Monotone) and Label substitution information
indeed also gives the correct Inverted part for inverted rules.
This is now checked by a new simple test in  test/joshua/decoder/SoftSyntacticConstraintDecodingTest.java that
checks that the "label substitution features with inversion" are fired as expected.