Dropped useless entries from class path.
Ignore some test files.
Added iterator access to a trie node's extensions, split by NTs and terminals.
Bugfix in child iterator.
Added safeguard.
Gideon : Added some info output and Assertions to DotChart (1)
as a first step to extending the code to implement soft constraints.
These assertions will later be again removed, but now served to
check wether the assumptions about were terminals and were nonterminals
are added are in fact true. For testing with Assert statements to work
in a java pipeline call, we needed also to have junit to be available
in th dependencies, build file and decoder, which is were the changes to
(2,3,4) come in .

  (1)    modified:   bin/joshua-decoder
  (2)    modified:   build.xml
  (3)    modified:   lib/ivy.xml
  (4)    modified:   src/joshua/decoder/chart_parser/DotChart.java
Gideon : Added a new test SoftSyntacticConstraintDecodingTest.java
that tests if soft syntactic constraint decoding matches, by checking if
the alternate derivations that are expected from soft matching of labels
are indeed generated. Currently this test still fails, as the new functionality is not
yet implemented. We are now thus well prepared to implement the new functionality.

As part of the effort to implement this new test, we also refactored the old
test code of ArtificialGrammarAndCorpusCreater and added new supporting classes
TestConfigFileCreater.java and ArtificialGrammarAndCorpusCreater.java that take care
of producing the required test files automatically.
Gideon : Refactored ../src/joshua/decoder/chart_parser/DotChart.java
by making the logic concerning the matching with respect to the boolean
"regExpMatching" produce a list of child_tnodes - or just one node if the boolean is set to false -
and have a generic block work with that list, rather than having an if/else block with a lot
of duplication were the only thing different is wether just one or a whole list of tnodes is produced
and processed.

This prepares for the next step: Creating a RuleMatcher class, that determines the matching behavior,
i.e. refactoring to strategy and getting rid of the boolean flag.
Gideon : Implemented a first working version of soft constraint Rule matching.
The new Interface RuleMatcher and implementation class RuleMatcherFactory
introduces methods that match rules, depending on whether what is matched is a non-terminal
or a terminal.
There are several implementations added, depending on whether regular expression matching is allowed and
whether soft syntactic constraint matching is allowed.
The test SoftSyntacticConstraintDecodingTest.java tests for a simple test grammar if the
right number of derivations is created, corresponding to alternate substitutions for the labels
of terminal rules.

       modified:   src/joshua/decoder/chart_parser/Chart.java
       modified:   src/joshua/decoder/chart_parser/DotChart.java
       new file:   src/joshua/decoder/chart_parser/RuleMatcher.java
       new file:   src/joshua/decoder/chart_parser/RuleMatcherFactory.java
       modified:   test/joshua/decoder/ArtificialGrammarAndCorpusCreater.java
       modified:   test/joshua/decoder/SoftSyntacticConstraintDecodingTest.java
       modified:   test/joshua/decoder/TestConfigFileCreater.java
Gideon : Extended the SoftSyntacticConstraintDecodingTest so it tests both normal decoding and decoding with soft syntactic constraints.
Adapted the JoshuaConfiguration so that it now contains a property that can be used to switch soft syntactic constraints decoding
on. Made some other minor improvements / clean ups of the code.

       modified:   src/joshua/decoder/JoshuaConfiguration.java
       modified:   src/joshua/decoder/chart_parser/Chart.java
       modified:   src/joshua/decoder/chart_parser/DotChart.java
       modified:   src/joshua/decoder/chart_parser/RuleMatcherFactory.java
       modified:   test/joshua/decoder/FeatureFunctionsTest.java
       modified:   test/joshua/decoder/SoftSyntacticConstraintDecodingTest.java
       modified:   test/joshua/decoder/TestConfigFileCreater.java
Gideon : Added some documentation explaining broadly the idea behind the new classes implementing Soft Syntactic Constraint Decoding
Gideon : Refactored the RuleMatching behavior in DotChart.java,
in reaction to feedback by Matt Post.
The nonterminal matching should only apply to the nonterminal level.
This simplifies what is required from RuleMatcher : it should only be concerned
with NonterminalMatcher.

For this reason I replaced RuleMatcher and RuleMatcherFactory
with a simpler interface and class
NonterminalMatcher and NonterminalMatcherFactory.
At the monent there are only two types of matching: strict matching and
soft syntactic matching (any nonterminal can substitute for any other
nonterminal, with the exceptions of OOV and GOAL which can only substitute
for themselves).
It is conceivable though that we want to implement other more fine grained
Nonterminal matchers, allowing only some types of nonterminals to match with
some other types perhaps?
For this reason I though it is still cleaner to add a interface and factory
method that implements this behavior, rather than making this all part of
the core class DotChart. The latter class is very fundamental in the  the
decoding process and should not be concerned with the behavior of what
types of nonterminals should match each other, I belief. But were to
put what code and how much to make use of strategy is also a matter of
taste I admit.
Gideon : Fixed a inefficiency / arguably potential bug in NonterminalMatcherFactory.java.
Basically the OOV symbol, should, just like the goal symbol, only be allowd to match to itself,
and also no other symbol should be allowed to match to it. This was previously not the case.
This has been fixed now by introducing the new boolean function
'isOOVLabelOrGoalLabel'. Firstly, for the label that is matched to
in the function 'produceMatchingChildTNodesNonterminalLevel'
if this function returns true, only strict matching is allowed.
Secondly, in the function matchAllEqualOrBothNonTerminalAndNotGoalOrOOV,
the function is applied that non-exact matching is only allowed for non-terminals
that are not OOV or GOAL symbols.

	modified:   src/joshua/decoder/chart_parser/NonterminalMatcherFactory.java
Gideon : Optimized the method  matchAllEqualOrBothNonTerminalAndNotGoalOrOOV in
 src/joshua/decoder/chart_parser/NonterminalMatcherFactory.java by removing unnescessary String matches:
        - Check for equal, this method will not be
        called when equality is required)
        - Check that wordID is a nonterminal. This is assured in the way the method is used
          and should not be used inside the loop
        (But a cheaper check that throws an exception outside the loop has still been maintained
        , to be safe, and make the assumtpions explicit)

        - Also the getkeys, then get item for keys has been replaced by a method that works
        with the EntrySet directly, avoiding separate extra lookup of every nonterminal in
        the HashSet, as we already loop over all of them anyway

 modified:   src/joshua/decoder/chart_parser/NonterminalMatcherFactory.java
Gideon : Implemented a smarter way to do the matching of alternate nonterminals
for soft syntactic constraint decoding. As it turns out, naive matching based on
looping over all child non-terminals in a children table from the Trie, is just
too expensive for reasonable size grammars.
In fact the best would be split the Trie in two parts: one for children that are non-terminals
and one for children that are terminals.
This would be the most general solution.
This is an expensive solution only in that is requires a thorough refactoring of the
rule table code.

As an intermediate solution we implemented first something now that works well for
small sets of Nonterminals. Rather than looping over all children and checking for
each if it is a nonterminal, we rather get a fixed set of present nonterminals
(except for OOV and Goal) in the grammar, and use those to directly get those
alternate nonterminals if present using the 'containsKey' and 'get' method of HashMap.
This is still relatively very cheap compared to the old alternative, provided the
number of Nonterminals is small. If the latter is not the case this solution will  not
work, we need to refactor the code and make an efficient method getNonterminalChildren
in the Trie implementation to deal with the more general case.
Gideon : Removed superflous log output from src/joshua/decoder/chart_parser/NonterminalMatcherFactory.java

 modified:   src/joshua/decoder/chart_parser/NonterminalMatcherFactory.java
Gideon : Minor fix to NonterminalMatcherfactory which should use
joshuaConfiguration.default_non_terminal to get the identity of the
default-nonterminal / OOV symbol, rather than a hard-coded [OOV] String.
The default-nonterminal is used in unary or glue rules to be able to produce
unknown words, but to avoid spurious ambiguity or weird unary chains
when performing soft syntactic decoding, it should not be allowed to
match or be matched to anything but itself, just like for the goal symbol.

	modified:   src/joshua/decoder/chart_parser/NonterminalMatcherFactory.java
Gideon : Adapted the format to the joshua conventions for the new test files
       modified:   test/joshua/decoder/ArtificialGrammarAndCorpusCreater.java
       modified:   test/joshua/decoder/FeatureFunctionsTest.java
       modified:   test/joshua/decoder/TestConfigFileCreater.java
Gideon :Refactoring Fuzzy Matching / Soft Syntactic Constraint Decoding code.
Replaced construction with NonterminalMatcher as interface with
NonterminalMatcherFactory.java as class providing implementations, with a simpler
construction with NonterminalMatcher as abstract method with different implementations for the
method

	public abstract List<Trie> produceMatchingChildTNodesNonterminalLevel(DotNode dotNode,
	    SuperNode superNode);

       modified:   src/joshua/decoder/chart_parser/Chart.java
       modified:   src/joshua/decoder/chart_parser/DotChart.java
       modified:   src/joshua/decoder/chart_parser/NonterminalMatcher.java
       deleted:    src/joshua/decoder/chart_parser/NonterminalMatcherFactory.java
       modified:   test/joshua/decoder/SoftSyntacticConstraintDecodingTest.java
Gideon : Replaced "softSyntacticConstraintDecoding"  with "fuzzy_matching" in the JoshuaConfiguration file.
Part of the effort to refactor the soft constraint decoding code to be compatible with the
Joshua conventions and preferences.

	modified:   src/joshua/decoder/JoshuaConfiguration.java
       modified:   src/joshua/decoder/chart_parser/NonterminalMatcher.java