Added missing test file (fixes #141)
Formatting and readability
Moved sentence filtering to its own file
Bugfix: now subtracting out the BLEU score from the final model cost.
Dropped useless entries from class path.
Merge branch 'master' of github.com:joshua-decoder/joshua into devel

Conflicts:
	.classpath
Removed useless entry from class path.
Ignore some test files.
Added iterator access to a trie node's extensions, split by NTs and terminals.
Bugfix in child iterator.
Added safeguard.
Gideon : Added some info output and Assertions to DotChart (1)
as a first step to extending the code to implement soft constraints.
These assertions will later be again removed, but now served to
check wether the assumptions about were terminals and were nonterminals
are added are in fact true. For testing with Assert statements to work
in a java pipeline call, we needed also to have junit to be available
in th dependencies, build file and decoder, which is were the changes to
(2,3,4) come in .

  (1)    modified:   bin/joshua-decoder
  (2)    modified:   build.xml
  (3)    modified:   lib/ivy.xml
  (4)    modified:   src/joshua/decoder/chart_parser/DotChart.java
Gideon : Added a new test SoftSyntacticConstraintDecodingTest.java
that tests if soft syntactic constraint decoding matches, by checking if
the alternate derivations that are expected from soft matching of labels
are indeed generated. Currently this test still fails, as the new functionality is not
yet implemented. We are now thus well prepared to implement the new functionality.

As part of the effort to implement this new test, we also refactored the old
test code of ArtificialGrammarAndCorpusCreater and added new supporting classes
TestConfigFileCreater.java and ArtificialGrammarAndCorpusCreater.java that take care
of producing the required test files automatically.
Gideon : Refactored ../src/joshua/decoder/chart_parser/DotChart.java
by making the logic concerning the matching with respect to the boolean
"regExpMatching" produce a list of child_tnodes - or just one node if the boolean is set to false -
and have a generic block work with that list, rather than having an if/else block with a lot
of duplication were the only thing different is wether just one or a whole list of tnodes is produced
and processed.

This prepares for the next step: Creating a RuleMatcher class, that determines the matching behavior,
i.e. refactoring to strategy and getting rid of the boolean flag.
Gideon : Implemented a first working version of soft constraint Rule matching.
The new Interface RuleMatcher and implementation class RuleMatcherFactory
introduces methods that match rules, depending on whether what is matched is a non-terminal
or a terminal.
There are several implementations added, depending on whether regular expression matching is allowed and
whether soft syntactic constraint matching is allowed.
The test SoftSyntacticConstraintDecodingTest.java tests for a simple test grammar if the
right number of derivations is created, corresponding to alternate substitutions for the labels
of terminal rules.

       modified:   src/joshua/decoder/chart_parser/Chart.java
       modified:   src/joshua/decoder/chart_parser/DotChart.java
       new file:   src/joshua/decoder/chart_parser/RuleMatcher.java
       new file:   src/joshua/decoder/chart_parser/RuleMatcherFactory.java
       modified:   test/joshua/decoder/ArtificialGrammarAndCorpusCreater.java
       modified:   test/joshua/decoder/SoftSyntacticConstraintDecodingTest.java
       modified:   test/joshua/decoder/TestConfigFileCreater.java
Gideon : Extended the SoftSyntacticConstraintDecodingTest so it tests both normal decoding and decoding with soft syntactic constraints.
Adapted the JoshuaConfiguration so that it now contains a property that can be used to switch soft syntactic constraints decoding
on. Made some other minor improvements / clean ups of the code.

       modified:   src/joshua/decoder/JoshuaConfiguration.java
       modified:   src/joshua/decoder/chart_parser/Chart.java
       modified:   src/joshua/decoder/chart_parser/DotChart.java
       modified:   src/joshua/decoder/chart_parser/RuleMatcherFactory.java
       modified:   test/joshua/decoder/FeatureFunctionsTest.java
       modified:   test/joshua/decoder/SoftSyntacticConstraintDecodingTest.java
       modified:   test/joshua/decoder/TestConfigFileCreater.java
Gideon : Added some documentation explaining broadly the idea behind the new classes implementing Soft Syntactic Constraint Decoding
Gideon : Refactored the RuleMatching behavior in DotChart.java,
in reaction to feedback by Matt Post.
The nonterminal matching should only apply to the nonterminal level.
This simplifies what is required from RuleMatcher : it should only be concerned
with NonterminalMatcher.

For this reason I replaced RuleMatcher and RuleMatcherFactory
with a simpler interface and class
NonterminalMatcher and NonterminalMatcherFactory.
At the monent there are only two types of matching: strict matching and
soft syntactic matching (any nonterminal can substitute for any other
nonterminal, with the exceptions of OOV and GOAL which can only substitute
for themselves).
It is conceivable though that we want to implement other more fine grained
Nonterminal matchers, allowing only some types of nonterminals to match with
some other types perhaps?
For this reason I though it is still cleaner to add a interface and factory
method that implements this behavior, rather than making this all part of
the core class DotChart. The latter class is very fundamental in the  the
decoding process and should not be concerned with the behavior of what
types of nonterminals should match each other, I belief. But were to
put what code and how much to make use of strategy is also a matter of
taste I admit.
Gideon : Fixed a inefficiency / arguably potential bug in NonterminalMatcherFactory.java.
Basically the OOV symbol, should, just like the goal symbol, only be allowd to match to itself,
and also no other symbol should be allowed to match to it. This was previously not the case.
This has been fixed now by introducing the new boolean function
'isOOVLabelOrGoalLabel'. Firstly, for the label that is matched to
in the function 'produceMatchingChildTNodesNonterminalLevel'
if this function returns true, only strict matching is allowed.
Secondly, in the function matchAllEqualOrBothNonTerminalAndNotGoalOrOOV,
the function is applied that non-exact matching is only allowed for non-terminals
that are not OOV or GOAL symbols.

	modified:   src/joshua/decoder/chart_parser/NonterminalMatcherFactory.java
Gideon : Optimized the method  matchAllEqualOrBothNonTerminalAndNotGoalOrOOV in
 src/joshua/decoder/chart_parser/NonterminalMatcherFactory.java by removing unnescessary String matches:
        - Check for equal, this method will not be
        called when equality is required)
        - Check that wordID is a nonterminal. This is assured in the way the method is used
          and should not be used inside the loop
        (But a cheaper check that throws an exception outside the loop has still been maintained
        , to be safe, and make the assumtpions explicit)

        - Also the getkeys, then get item for keys has been replaced by a method that works
        with the EntrySet directly, avoiding separate extra lookup of every nonterminal in
        the HashSet, as we already loop over all of them anyway

 modified:   src/joshua/decoder/chart_parser/NonterminalMatcherFactory.java