silk-framework/silk — pull request #29 by ziodave (2014-11-14 14:06:11+00:00)
Title: add support for executing transformations in singlemachine; add logging; other minor changes

Hello,

**Executing Transformations from *singlemachine***

I made an update to Silk in order to support executing transformations from singlemachine in addition to linkings.

Taking inspiration from _linking_, the update consists in:

* adding a TransformationSpecification class which holds all the required information for transformations,
* updating the Silk LSL XSD in order to allow configuring Transforms/Transform/TransformRule(s),
* adding support for TransformationSpecification and their executing in LinkingConfig,
* allow execution of all Transformations (or by the specified Identifier) from Silk.scala.

Files added/changed:

silk-core/src/main/scala/de/fuberlin/wiwiss/silk/config/TransformSpecification.scala
silk-core/src/main/resources/de/fuberlin/wiwiss/silk/LinkSpecificationLanguage.xsd
silk-core/src/main/scala/de/fuberlin/wiwiss/silk/config/LinkingConfig.scala
silk-core/src/main/scala/de/fuberlin/wiwiss/silk/execution/ExecuteTransform.scala
silk-tools/silk-singlemachine/src/main/scala/de/fuberlin/wiwiss/silk/Silk.scala

**Add more debug logging**

To help during development/testing I added more _FINE_ logging to the following files:

silk-core/src/main/scala/de/fuberlin/wiwiss/silk/cache/FileEntityCache.scala
silk-core/src/main/scala/de/fuberlin/wiwiss/silk/config/Blocking.scala
silk-core/src/main/scala/de/fuberlin/wiwiss/silk/execution/LoadTask.scala
silk-core/src/main/scala/de/fuberlin/wiwiss/silk/plugins/CorePlugins.scala
silk-core/src/main/scala/de/fuberlin/wiwiss/silk/plugins/dataset/csv/CsvSource.scala
silk-core/src/main/scala/de/fuberlin/wiwiss/silk/plugins/dataset/xml/XmlSource.scala
silk-core/src/main/scala/de/fuberlin/wiwiss/silk/runtime/plugin/PluginFactory.scala
silk-plugins/silk-plugins-jena/src/main/scala/de/fuberlin/wiwiss/silk/plugins/dataset/JenaPlugins.scala
silk-plugins/silk-plugins-jena/src/main/scala/de/fuberlin/wiwiss/silk/plugins/dataset/rdf/RdfDataSource.scala

**CSV line regex update**

I made a small change with the hope of improving a bit the performance, by compiling the regex before using it:

silk-core/src/main/scala/de/fuberlin/wiwiss/silk/plugins/dataset/csv/CsvSource.scala

**Things that require particular attention**

I made also the following two changes, please check them:

* remove unused `globalThreshold: Option[Double]` as I couldn't find it used anywhere (but maybe I am mistaken?)

silk-core/src/main/scala/de/fuberlin/wiwiss/silk/linkagerule/TransformRule.scala

* change `!currentStatus.isInstanceOf[TaskCanceling] || status.isInstanceOf[TaskFinished]` to `!currentStatus.isInstanceOf[TaskCanceling]`:

silk-core/src/main/scala/de/fuberlin/wiwiss/silk/runtime/task/HasStatus.scala

I am at your disposal in case you need further information.

Cheers,
David