Initialization findlinks and webcrawl search - flcr
Contains: ParameterParser, IOUtil, LanguageParser,
LanguageDependentMover and LanguageDirectoryMover. For all parts JUnit
tests exists.
Setting the first published round of findlinks and webcrawl search - flcr - Version 0.0.1
First step of iterable development: splitted unit tests to integration and unit tests. Now, AllIntegrationTests exists next to AllUnitTests. This way, all Integration tests are extending their unit tests. Everything else should be placed inside AllUnitTests. Therefore a few tests will run twice during deployment. However, more runs of tests may just show some errors, which will still exist. The copying of resources is now working. However, the further tests have to be done, e.g. in CopyControllerIntegrationTest. A test is required for splitting a resource into more than one file depending on the assigned property file containing a number of domains, which should be seperated.
Commiting domain splitting feature. Unit tests and integration tests are not applied to the new environment. By assigning a property file with language and domain patterns, input files are splitted accordingly. Now, a refactoring has to start, and the adaption of the functionality to the command line interface (view component).
Reworked test classes for a better logging and a better structure for the tests.
Added a new feature for evaluating the results of Consumer and Producer controlled by the CopyController. A BlockingQueue instance with results is shared between the threads. A value 'false' marks an unrecognized exception, otherwise the CommandExecutionException is thrown, which has now a Constructor for String messages. IntegrationTestCase changed because the underlaying Consumer and Producer Constructor requiring the new result blocking queue
Small change, but with a bigger impact. the feature 'generalLanguageAllowed' is removed. It was not required, and is not wished. It would create ambigous behaviour of the CopyController. The UnitTeest had to change respectivally
IOUtil supports a new feature: collecting files (recursivly if wished) of an assigned directory file path
CopyManager solution 0: working with incomplete barrier combining every producer (only one producer for every textfile instance) with its consumers (depending on the number of cores and on the local task queue of producer)
Lets commit first: there are so many bugs left, and so many work-arounds in this package. This commit alone brings so many different classes into the work tree, which are probably not required anymore. Therefore this commit can be seen as a first clean-up. This marks version 1.0git stgit st! Therefore a first stable version can be published with this commit. The only supported parameter configuration yet: 'jar --input DIRECTORY --output DIRECTORY --no-threads > output.logs' Especially the threads have still problems: ComplexCopyManager. Another thing is the recognition of valid and invalid input files: files like FL____.txt are still recognized as valid. How about files like FL_0000.txt etc? Starting a BUG list now, and keeping it in the working tree. Please see there for more details, and extend the languagedomain.conf for more domains, which are of interest.
BugFix of concurrent problem of ComplexCopyManager (previously ComplexCopyController). The bug was a limitless task queue between the producers and consumers. Enabled the ComplexCopyManager and SimpleCopyManager to allow single file assignments. Therefore a better testability should be given. A new features was started too: allowing sources without a pre-defined language ('___' files) which should be mapped to 'NO_LANGUAGE' language. However the resulting textfiles are without any language and domain information. This bug should be fixed probably soon.
BugFIX: Now no-language files are processed too. The Bug was in the detail: all used pattern were lowercase pattern like [a-z0-9], the default value for no-language: NO_LANGUAGE. therefore the language was not recognized as such, and a almost empty file was created. The sourcedomainmapping and defaultMapping functions of MappingFactory should work now. Hopefully no other bugs were opened with the taken actions. SourceInputStreamPool has now a more reliable validation: checking content and language. This version marks also 1.02 stable release
Bringing this application to github
Clean up of the repository for sharing it with the whole world.
Clean up of the repository for sharing it with the whole world.