Repository: OpenRefine/OpenRefine (java)

Issue:
Custom clustering function with non-GREL expressions
<!-- Describe the bug - Please add a clear and concise description of the bug above this line. You can delete this line if you want. It will be hidden in the final bug report -->

### To Reproduce

Steps to reproduce the behavior:

1. Register a custom clustering function with either a Jython `return 1` or Clojure `(float 1)` expression.
2. Then try to use the custom clustering function.

### Current Results

For Clojure:

```log
com.google.refine.expr.ParsingException: Parsing error at offset 7: Missing )
        at com.google.refine.grel.Parser.makeException(Parser.java:93)
        at com.google.refine.grel.Parser.parseFactor(Parser.java:250)
        at com.google.refine.grel.Parser.parseTerm(Parser.java:146)
        at com.google.refine.grel.Parser.parseSubExpression(Parser.java:124)
        at com.google.refine.grel.Parser.parseExpression(Parser.java:100)
        at com.google.refine.grel.Parser.<init>(Parser.java:79)
        at com.google.refine.grel.Parser.<init>(Parser.java:72)
        at com.google.refine.grel.Parser$1.parse(Parser.java:62)
        at com.google.refine.expr.MetaParser.parse(MetaParser.java:117)
        at com.google.refine.clustering.binning.UserDefinedKeyer.<init>(UserDefinedKeyer.java:48)
        ...
```

For Jython:

```log
java.lang.IllegalArgumentException: The object is null
        at com.google.refine.clustering.knn.UserDefinedDistance.convertToDouble(UserDefinedDistance.java:76)
        at com.google.refine.clustering.knn.UserDefinedDistance.compute(UserDefinedDistance.java:71)
        at com.google.refine.clustering.knn.kNNClusterer$BlockingClusteringRowVisitor$DistanceWrapper.d(kNNClusterer.java:192)
        at edu.mit.simile.vicino.clustering.NGramClusterer$BlockEvaluator.call(NGramClusterer.java:78)
        at edu.mit.simile.vicino.clustering.NGramClusterer$BlockEvaluator.call(NGramClusterer.java:51)
        ...
```

### Expected Behavior

Being able to use non-GREL expressions with custom clustering functions.

### Versions
 - Operating System: Windows 10
 - Browser Version: Firefox 136
 - JRE or JDK Version: embedded
 - OpenRefine: 3.9.2

### Additional context

In [UserDefinedKeyer.java](https://github.com/OpenRefine/OpenRefine/blob/930b303376b31a02fc7375c1b994a4ad501bd48d/main/src/com/google/refine/clustering/binning/UserDefinedKeyer.java#L48) and [UserDefinedDistance.java](https://github.com/OpenRefine/OpenRefine/blob/930b303376b31a02fc7375c1b994a4ad501bd48d/main/src/com/google/refine/clustering/knn/UserDefinedDistance.java#L49) the Metaparser is used as ` MetaParser.parse("grel:" + expression)`. So it seems that the restriction to GREL expressions for custom clustering functions is hardcoded.


Pull request:
Enable custom clustering for non-GREL expressions
Fixes #7234 

In #6612 we introduced custom clustering functions.
But with non-GREL expressions like Jython and Clojure the clustering failed on multiple levels.

1. Several post requests in the new clustering dialogs were missing csrf tokens.
2. Several post requests did only sent the plain expression without the language code.
3. The GREL language code was hard coded in `UserDefinedKeyer.java` and `UserDefinedDistance.java`.
4. The new variables `value1` and `value2` were not made available in Jython and Clojure expressions.

The last one is probably the most problematic.

In GREL all variables in `bindings` are basically available as globals. But in Jython and Clojure we encapsulate the expressions in functions and only pass certain variables as parameters (`value, cell, cells, row, rowIndex`).

This means that in Jython and Clojure expressions we can not access variables like `columnName`. This also means that we somehow have to make the new variables `value1` and `value2` accessible to Jython and Clojure expressions.

Without rewriting the whole Jython and Clojure integration it felt most idiomatic to also introduce the new variables as parameters to the Jython and Clojure functions that encapsulate the expressions. But it is still somewhat unsatisfying that we have a different pool of available variables in GREL and (possibly) each other language integration.

**Note:** for the custom clustering a separate set of `bindings` is created. This means we basically only have access to `value` and `value1, value2` respectively. Usually available variables like `cell, cells, row, rowIndex, ...` are not available (not even in GREL). As this seems to be a deliberate choice I did not touch this part. But we should also document this behavior e.g. on [Custom clustering methods](https://openrefine.org/docs/manual/cellediting#custom-clustering-methods) to avoid further confusion.

Patch:
diff --git a/extensions/jython/src/com/google/refine/jython/JythonEvaluable.java b/extensions/jython/src/com/google/refine/jython/JythonEvaluable.java
index 4e536fcbf..77c7228cb 100644
--- a/extensions/jython/src/com/google/refine/jython/JythonEvaluable.java
+++ b/extensions/jython/src/com/google/refine/jython/JythonEvaluable.java
@@ -133,7 +133,7 @@ public class JythonEvaluable implements Evaluable {
         StringBuffer sb = new StringBuffer(1024);
         sb.append("def ");
         sb.append(s_functionName);
-        sb.append("(value, cell, cells, row, rowIndex):");
+        sb.append("(value, cell, cells, row, rowIndex, value1, value2):");
         for (String line : lines) {
             sb.append("\n  ");
             sb.append(line);
@@ -145,23 +145,17 @@ public class JythonEvaluable implements Evaluable {
     @Override
     public Object evaluate(Properties bindings) {
         try {
-            Object value = bindings.get("value");
-            PyObject pyValue;
-            if (value instanceof OffsetDateTime) {
-                // We know the OffsetDateTime is always at UTC, but just in case it ever changes, do the UTC change
-                pyValue = Py.newDatetime(Timestamp.valueOf(((OffsetDateTime) value).atZoneSameInstant(ZoneOffset.UTC).toLocalDateTime()));
-            } else {
-                pyValue = Py.java2py(value);
-            }
             // call the temporary PyFunction directly
             Object result = ((PyFunction) _engine.get(s_functionName)).__call__(
 
                     new PyObject[] {
-                            pyValue,
-                            new JythonHasFieldsWrapper((HasFields) bindings.get("cell"), bindings),
-                            new JythonHasFieldsWrapper((HasFields) bindings.get("cells"), bindings),
-                            new JythonHasFieldsWrapper((HasFields) bindings.get("row"), bindings),
-                            Py.java2py(bindings.get("rowIndex"))
+                            getValue("value", bindings),
+                            getObject("cell", bindings),
+                            getObject("cells", bindings),
+                            getObject("row", bindings),
+                            getValue("rowIndex", bindings),
+                            getValue("value1", bindings),
+                            getValue("value2", bindings)
                     });
 
             return unwrap(result);
@@ -170,6 +164,24 @@ public class JythonEvaluable implements Evaluable {
         }
     }
 
+    private JythonHasFieldsWrapper getObject(String key, Properties bindings) {
+        return new JythonHasFieldsWrapper((HasFields) bindings.get(key), bindings);
+    }
+
+    private PyObject getValue(String key, Properties bindings) {
+        Object value = bindings.get(key);
+        PyObject pyValue;
+        if (value instanceof OffsetDateTime) {
+            // We know the OffsetDateTime is always at UTC, but just in case it ever
+            // changes, do the UTC change
+            pyValue = Py.newDatetime(
+                    Timestamp.valueOf(((OffsetDateTime) value).atZoneSameInstant(ZoneOffset.UTC).toLocalDateTime()));
+        } else {
+            pyValue = Py.java2py(value);
+        }
+        return pyValue;
+    }
+
     protected Object unwrap(Object result) {
         if (result != null) {
             if (result instanceof JythonObjectWrapper) {
diff --git a/main/src/com/google/refine/clustering/binning/UserDefinedKeyer.java b/main/src/com/google/refine/clustering/binning/UserDefinedKeyer.java
index f62c3789e..b0e5a4593 100644
--- a/main/src/com/google/refine/clustering/binning/UserDefinedKeyer.java
+++ b/main/src/com/google/refine/clustering/binning/UserDefinedKeyer.java
@@ -45,7 +45,7 @@ public class UserDefinedKeyer extends Keyer {
     private Properties bindings;
 
     public UserDefinedKeyer(String expression) throws ParsingException {
-        eval = MetaParser.parse("grel:" + expression);
+        eval = MetaParser.parse(expression);
         bindings = new Properties();
 
         bindings.put("true", true);
diff --git a/main/src/com/google/refine/clustering/knn/UserDefinedDistance.java b/main/src/com/google/refine/clustering/knn/UserDefinedDistance.java
index 43fee345b..4971abe29 100644
--- a/main/src/com/google/refine/clustering/knn/UserDefinedDistance.java
+++ b/main/src/com/google/refine/clustering/knn/UserDefinedDistance.java
@@ -46,7 +46,7 @@ public class UserDefinedDistance implements SimilarityDistance {
     private Properties bindings;
 
     public UserDefinedDistance(String expression) throws ParsingException {
-        eval = MetaParser.parse("grel:" + expression);
+        eval = MetaParser.parse(expression);
         bindings = new Properties();
 
         bindings.put("true", true);
diff --git a/main/src/com/google/refine/expr/ClojureParser.java b/main/src/com/google/refine/expr/ClojureParser.java
index b20f2054d..0d98978b0 100644
--- a/main/src/com/google/refine/expr/ClojureParser.java
+++ b/main/src/com/google/refine/expr/ClojureParser.java
@@ -50,7 +50,7 @@ public class ClojureParser implements LanguageSpecificParser {
 //                    RT.load("clojure/core"); // Make sure RT is initialized
             Object foo = RT.CURRENT_NS; // Make sure RT is initialized
             IFn fn = (IFn) clojure.lang.Compiler.load(new StringReader(
-                    "(fn [value cell cells row rowIndex] " + source + ")"));
+                    "(fn [value cell cells row rowIndex value1 value2] " + source + ")"));
 
             // TODO: We should to switch from using Compiler.load
             // because it's technically an internal interface
@@ -75,7 +75,9 @@ public class ClojureParser implements LanguageSpecificParser {
                                 bindings.get("cell"),
                                 bindings.get("cells"),
                                 bindings.get("row"),
-                                bindings.get("rowIndex"));
+                                bindings.get("rowIndex"),
+                                bindings.get("value1"),
+                                bindings.get("value2"));
                     } catch (Exception e) {
                         return new EvalError(e.getMessage());
                     }
diff --git a/main/webapp/modules/core/scripts/dialogs/clustering-dialog.js b/main/webapp/modules/core/scripts/dialogs/clustering-dialog.js
index 9dff926e3..dde1a290b 100644
--- a/main/webapp/modules/core/scripts/dialogs/clustering-dialog.js
+++ b/main/webapp/modules/core/scripts/dialogs/clustering-dialog.js
@@ -226,7 +226,7 @@ ClusteringDialog.prototype._renderClusteringFunctions = function() {
                     var option = $('<option></option>')
                         .val("UserDefinedKeyer")
                         .text(functions[i].name)
-                        .data('expression', functions[i].expression)
+                        .data('expression', functions[i].expressionLang + ':' + functions[i].expression)
                         .appendTo(self._elmts.keyingFunctionSelector);
 
                 }
@@ -243,7 +243,7 @@ ClusteringDialog.prototype._renderClusteringFunctions = function() {
                     var option = $('<option></option>')
                         .val("UserDefinedDistance")
                         .text(functions[i].name)
-                        .data('expression', functions[i].expression)
+                        .data('expression', functions[i].expressionLang + ':' + functions[i].expression)
                         .appendTo(self._elmts.distanceFunctionSelector);
                 }
             },
diff --git a/main/webapp/modules/core/scripts/dialogs/clustering-functions-dialog.js b/main/webapp/modules/core/scripts/dialogs/clustering-functions-dialog.js
index 01654af29..aef4c8adb 100644
--- a/main/webapp/modules/core/scripts/dialogs/clustering-functions-dialog.js
+++ b/main/webapp/modules/core/scripts/dialogs/clustering-functions-dialog.js
@@ -223,7 +223,7 @@ ClusteringFunctionsDialog.prototype._addFunction = function (column) {
                             }),
                             data: {
                                 "value": JSON.stringify(_functions),
-                                csrf_token: token
+                                "csrf_token": token
                             },
                             success: function (data) {
                                 self._renderTable();
@@ -313,7 +313,7 @@ ClusteringFunctionsDialog.prototype._editFunction = function (column, functionsT
                             }),
                             data: {
                                 "value": JSON.stringify(_functions),
-                                csrf_token: token
+                                "csrf_token": token
                             },
                             success: function (data) {
                                 self._renderTable();
@@ -352,7 +352,7 @@ ClusteringFunctionsDialog.prototype._deleteFunction = function (index) {
                             }),
                             data: {
                                 "value": JSON.stringify(_functions),
-                                csrf_token: token
+                                "csrf_token": token
                             },
                             success: function (data) {
                                 self._renderTable();
diff --git a/main/webapp/modules/core/scripts/dialogs/expression-preview-dialog.js b/main/webapp/modules/core/scripts/dialogs/expression-preview-dialog.js
index 8ee657061..ed8be922a 100644
--- a/main/webapp/modules/core/scripts/dialogs/expression-preview-dialog.js
+++ b/main/webapp/modules/core/scripts/dialogs/expression-preview-dialog.js
@@ -468,12 +468,12 @@ ExpressionPreviewDialog.Widget.prototype.update = function() {
         );
 
         self._params = {
-            "expression" : expression,
+            "expression" : this._getLanguage() + ":" + expression,
             "radius" : Number(document.getElementById('radius').value),
             "blocking-ngram-size" : Number(document.getElementById('blockingChars').value)
         };
 
-        $.post(
+        Refine.postCSRF(
             "command/core/compute-clusters?" + $.param({ project: theProject.id }),
             {
                 engine: JSON.stringify(ui.browsingEngine.getJSON()),
@@ -664,7 +664,7 @@ ExpressionPreviewDialog.Widget.prototype._renderDistancePreview = function(first
         let newExpression = expression.replace(/value1/g, '"' + value1.toString().replaceAll(' ', '\xa0') + '"')
                                 .replace(/value2/g, '"' + value2.toString().replaceAll(' ', '\xa0') + '"')
                                 .replace(/value/g,"");
-        $.post(
+        Refine.postCSRF(
             "command/core/preview-expression?" + $.param(params), 
             {
                 expression: self._getLanguage() + ":" + newExpression,


Test patch:
diff --git a/extensions/jython/tests/src/com/google/refine/jython/JythonEvaluableTest.java b/extensions/jython/tests/src/com/google/refine/jython/JythonEvaluableTest.java
index 37ea402f1..7fea363dc 100644
--- a/extensions/jython/tests/src/com/google/refine/jython/JythonEvaluableTest.java
+++ b/extensions/jython/tests/src/com/google/refine/jython/JythonEvaluableTest.java
@@ -2,6 +2,7 @@
 package com.google.refine.jython;
 
 import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertTrue;
 
 import java.time.OffsetDateTime;
 import java.time.ZoneOffset;
@@ -12,6 +13,7 @@ import org.testng.Assert;
 import org.testng.annotations.Test;
 
 import com.google.refine.expr.CellTuple;
+import com.google.refine.expr.EvalError;
 import com.google.refine.expr.Evaluable;
 import com.google.refine.expr.ParsingException;
 import com.google.refine.model.Cell;
@@ -149,4 +151,33 @@ public class JythonEvaluableTest {
     public void testParseError() throws ParsingException {
         JythonEvaluable.createParser().parse("foo(", "jython");
     }
+
+    @Test
+    public void testEvalErrorOnSomeBindings() {
+        String testKey = "columnName";
+        Properties bindings = createBindings();
+        assertTrue(bindings.containsKey(testKey), "Bindings should contain '" + testKey + "'.");
+
+        Evaluable evaluable = new JythonEvaluable("return " + testKey);
+        Object result = evaluable.evaluate(bindings);
+        assertTrue(result instanceof EvalError,
+                "Should produce an EvalError because '" + testKey + "' is not available.");
+        assertTrue(result.toString().contains(testKey),
+                "EvalError should contain the name '" + testKey + "' in the error message.");
+    }
+
+    @Test
+    public void testSpecialValuesForUserDefinedDistance() {
+        Properties bindings = createBindings();
+        String a = "aaaa";
+        String b = "bbb";
+        // special values used in custom clustering via UserDefinedDistance
+        bindings.put("value1", a);
+        bindings.put("value2", b);
+
+        long expectedLength = a.length() - b.length();
+        Evaluable evaluable = new JythonEvaluable("return len(value1) - len(value2)");
+        Object result = evaluable.evaluate(bindings);
+        assertEquals(result, expectedLength, "Length difference should be the same.");
+    }
 }
diff --git a/main/tests/server/src/com/google/refine/clustering/binning/UserDefinedKeyerTests.java b/main/tests/server/src/com/google/refine/clustering/binning/UserDefinedKeyerTests.java
index e1637ad93..0d50b030c 100644
--- a/main/tests/server/src/com/google/refine/clustering/binning/UserDefinedKeyerTests.java
+++ b/main/tests/server/src/com/google/refine/clustering/binning/UserDefinedKeyerTests.java
@@ -31,69 +31,110 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
 package com.google.refine.clustering.binning;
 
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.when;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import org.mockito.AdditionalAnswers;
+import org.mockito.Mock;
+import org.mockito.MockitoAnnotations;
 import org.testng.Assert;
 import org.testng.annotations.AfterMethod;
 import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.DataProvider;
 import org.testng.annotations.Test;
 
+import com.google.refine.expr.Evaluable;
+import com.google.refine.expr.LanguageSpecificParser;
 import com.google.refine.expr.MetaParser;
 import com.google.refine.expr.ParsingException;
 import com.google.refine.grel.Parser;
 
 public class UserDefinedKeyerTests {
 
-    private Keyer keyer;
+    private List<String> testStrings = List.of("hello", "new", "fantastic");
+
+    @Mock
+    private LanguageSpecificParser testParser;
+
+    @Mock
+    private Evaluable testEvaluable;
+
     // some sample test cases taken from KeyerTests
-    private static final String[][] testStrings = {
-            { " école ÉCole ecoLe ", "ecole" },
-            { " d c b a ", "a b c d" },
-            { "\tABC \t DEF ", "abc def" }, // test leading and trailing whitespace
-            { "bbb\taaa", "aaa bbb" },
-//        {"å","aa"}, // Requested by issue #650, but conflicts with diacritic folding
-            { "æø", "aeoe" }, // Norwegian replacements from #650
-            { "©ß", "css" }, // issue #409 esszet
-    };
+    @DataProvider(name = "user-defined-keyer-test-cases")
+    private String[][] userDefinedKeyerTestCases() {
+        return new String[][] {
+                { " école ÉCole ecoLe ", "ecole" },
+                { " d c b a ", "a b c d" },
+                { "\tABC \t DEF ", "abc def" }, // test leading and trailing whitespace
+                { "bbb\taaa", "aaa bbb" },
+                // {"å","aa"}, // Requested by issue #650, but conflicts with diacritic folding
+                { "æø", "aeoe" }, // Norwegian replacements from #650
+                { "©ß", "css" }, // issue #409 esszet
+        };
+    }
+
+    @DataProvider(name = "user-defined-keyer-expressions")
+    private String[][] userDefinedKeyerExpressions() {
+        return new String[][] {
+                { "value + ' world'", },
+                { "grel:value + ' world'", },
+                { "testparser:value + world" },
+        };
+    }
+
+    @BeforeMethod
+    public void setupMocks() throws ParsingException {
+        MockitoAnnotations.openMocks(this);
+        when(testParser.parse(anyString(), eq("testparser"))).thenReturn(testEvaluable);
+        when(testEvaluable.evaluate(any())).thenAnswer(AdditionalAnswers
+                .returnsElementsOf(testStrings.stream().map(s -> s + " world").collect(Collectors.toList())));
+    }
 
     @BeforeMethod
-    public void registerGRELParser() {
+    public void registerParser() {
         MetaParser.registerLanguageParser("grel", "GREL", Parser.grelParser, "value");
+        MetaParser.registerLanguageParser("testparser", "Test Parser", testParser, "return value");
     }
 
     @AfterMethod
-    public void unregisterGRELParser() {
+    public void unregisterParser() {
         MetaParser.unregisterLanguageParser("grel");
+        MetaParser.unregisterLanguageParser("testparser");
     }
 
-    @Test
-    public void testUserDefinedKeyer1() {
+    @Test(dataProvider = "user-defined-keyer-test-cases")
+    public void testGenericCases(String testString, String expectedString) {
+        UserDefinedKeyer keyer;
         try {
             String expression = "value.fingerprint()";
             keyer = new UserDefinedKeyer(expression);
         } catch (ParsingException e) {
             throw new RuntimeException(e);
         }
-        for (String[] ss : testStrings) {
-            Assert.assertEquals(ss.length, 2, "Invalid test"); // Not a valid test
-            Assert.assertEquals(keyer.key(ss[0]), ss[1],
-                    "User defined keying for string: " + ss[0] + " failed");
-        }
+        Assert.assertEquals(keyer.key(testString), expectedString,
+                "User defined keying for string: " + testString + " failed");
     }
 
-    @Test
-    public void testUserDefinedKeyer2() {
+    @Test(dataProvider = "user-defined-keyer-expressions")
+    public void testDifferentExpressions(String expression) {
+        UserDefinedKeyer keyer;
         try {
-            String expression = "value + ' world'";
             keyer = new UserDefinedKeyer(expression);
         } catch (ParsingException e) {
+            e.printStackTrace();
+            Assert.fail("Could not parse expression " + expression);
             throw new RuntimeException(e);
         }
 
-        String[] strs = { "hello", "new", "fantastic" };
-        for (String s : strs) {
+        for (String s : testStrings) {
             String output = keyer.key(s);
             Assert.assertEquals(output, s + " world",
                     "User defined keying for string: " + s + " failed");
         }
     }
-
 }
diff --git a/main/tests/server/src/com/google/refine/clustering/knn/DistanceTests.java b/main/tests/server/src/com/google/refine/clustering/knn/DistanceTests.java
index 6dbe37d3a..631774558 100644
--- a/main/tests/server/src/com/google/refine/clustering/knn/DistanceTests.java
+++ b/main/tests/server/src/com/google/refine/clustering/knn/DistanceTests.java
@@ -33,44 +33,74 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
 package com.google.refine.clustering.knn;
 
-import org.slf4j.LoggerFactory;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.when;
+
+import java.util.List;
+
+import org.apache.commons.lang3.StringUtils;
+import org.mockito.AdditionalAnswers;
+import org.mockito.Mock;
+import org.mockito.MockitoAnnotations;
 import org.testng.Assert;
 import org.testng.annotations.AfterMethod;
 import org.testng.annotations.BeforeMethod;
-import org.testng.annotations.BeforeTest;
+import org.testng.annotations.DataProvider;
 import org.testng.annotations.Test;
 
-import com.google.refine.RefineTest;
+import com.google.refine.expr.Evaluable;
+import com.google.refine.expr.LanguageSpecificParser;
 import com.google.refine.expr.MetaParser;
 import com.google.refine.expr.ParsingException;
 import com.google.refine.grel.Parser;
 
-public class DistanceTests extends RefineTest {
+public class DistanceTests {
+
+    private List<Integer> testLengths = List.of(1, 3, 5);
+
+    @Mock
+    private LanguageSpecificParser testParser;
 
-    private static SimilarityDistance _distance;
+    @Mock
+    private Evaluable testEvaluable;
 
-    @Override
-    @BeforeTest
-    public void init() {
-        logger = LoggerFactory.getLogger(this.getClass());
+    @DataProvider(name = "user-defined-distance-expressions")
+    private String[][] userDefinedKeyerExpressions() {
+        return new String[][] {
+                { "value1.length() - value2.length()", },
+                { "grel:value1.length() - value2.length()", },
+                { "testparser:value1.length() - value2.length()" },
+        };
     }
 
     @BeforeMethod
-    public void registerGRELParser() {
+    public void setupMocks() throws ParsingException {
+        MockitoAnnotations.openMocks(this);
+        when(testParser.parse(anyString(), eq("testparser"))).thenReturn(testEvaluable);
+        when(testEvaluable.evaluate(any())).thenAnswer(AdditionalAnswers.returnsElementsOf(testLengths));
+    }
+
+    @BeforeMethod
+    public void registerParser() {
         MetaParser.registerLanguageParser("grel", "GREL", Parser.grelParser, "value");
+        MetaParser.registerLanguageParser("testparser", "Test Parser", testParser, "return value");
     }
 
     @AfterMethod
-    public void unregisterGRELParser() {
+    public void unregisterParser() {
         MetaParser.unregisterLanguageParser("grel");
+        MetaParser.unregisterLanguageParser("testparser");
     }
 
     @Test
     public void testUserDefinedDistance() {
         String expression = "value1.length() - value2.length()";
 
+        SimilarityDistance distance;
         try {
-            _distance = new UserDefinedDistance(expression);
+            distance = new UserDefinedDistance(expression);
         } catch (ParsingException e) {
             throw new RuntimeException(e);
         }
@@ -78,9 +108,31 @@ public class DistanceTests extends RefineTest {
         String[] testString = { "Ahmed", "Mohamed", "Omar", "Othman", "Aly", "Eleraky" };
         for (int i = 0; i < testString.length; i++) {
             for (int j = i + 1; j < testString.length; j++) {
-                Assert.assertEquals(testString[i].length() - testString[j].length(), _distance.compute(testString[i], testString[j]),
+                Assert.assertEquals(testString[i].length() - testString[j].length(),
+                        distance.compute(testString[i], testString[j]),
                         "User defined distance for strings: " + testString[i] + ", " + testString[j] + " failed");
             }
         }
     }
+
+    @Test(dataProvider = "user-defined-distance-expressions")
+    public void testUserDefinedDistanceWithDifferentExpressions(String expression) {
+        SimilarityDistance distance;
+        try {
+            distance = new UserDefinedDistance(expression);
+        } catch (ParsingException e) {
+            e.printStackTrace();
+            Assert.fail("Could not parse expression " + expression);
+            throw new RuntimeException(e);
+        }
+
+        for (int expectedDistance : testLengths) {
+            int baseLength = 10;
+            String testString1 = StringUtils.repeat("*", baseLength);
+            String testString2 = StringUtils.repeat("*", baseLength - expectedDistance);
+            double computedDistance = distance.compute(testString1, testString2);
+            Assert.assertEquals(computedDistance, expectedDistance,
+                    "User defined distance for strings: " + testString1 + ", " + testString2 + " failed");
+        }
+    }
 }
diff --git a/main/tests/server/src/com/google/refine/expr/ClojureParserTest.java b/main/tests/server/src/com/google/refine/expr/ClojureParserTest.java
new file mode 100644
index 000000000..256bb85ba
--- /dev/null
+++ b/main/tests/server/src/com/google/refine/expr/ClojureParserTest.java
@@ -0,0 +1,73 @@
+
+package com.google.refine.expr;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertTrue;
+
+import java.math.BigDecimal;
+import java.util.Properties;
+
+import org.testng.annotations.Test;
+
+import com.google.refine.model.Cell;
+import com.google.refine.model.Project;
+import com.google.refine.model.Row;
+
+public class ClojureParserTest {
+
+    private Properties createBindings() {
+        Project project = new Project();
+        Row row = new Row(2);
+        row.setCell(0, new Cell("one", null));
+        row.setCell(0, new Cell("1", null));
+        Properties bindings = new Properties();
+        bindings.put("columnName", "number");
+        bindings.put("true", "true");
+        bindings.put("false", "false");
+        bindings.put("rowIndex", "0");
+        bindings.put("value", 1);
+        bindings.put("project", project);
+        bindings.put("call", "number");
+        bindings.put("PI", "3.141592654");
+        bindings.put("cells", new CellTuple(project, row));
+        return bindings;
+    }
+
+    @Test
+    public void testBasicClojureExpression() throws ParsingException {
+        Properties bindings = createBindings();
+        // just make sure that value is a number
+        bindings.put("value", 1);
+
+        Evaluable evaluable = (new ClojureParser()).parse("(+ 0 value)", "clojure");
+        Object result = evaluable.evaluate(bindings);
+        assertEquals(new BigDecimal((long) result), new BigDecimal((int) bindings.get("value")),
+                "Adding 0 to a number should keep the number.");
+    }
+
+    @Test(expectedExceptions = ParsingException.class)
+    public void testEvalErrorOnSomeBindings() throws ParsingException {
+        String testKey = "columnName";
+        Properties bindings = createBindings();
+        assertTrue(bindings.containsKey(testKey), "Bindings should contain '" + testKey + "'.");
+
+        Evaluable evaluable = (new ClojureParser()).parse("(count " + testKey + ")", "clojure");
+        evaluable.evaluate(bindings);
+    }
+
+    @Test
+    public void testSpecialValuesForUserDefinedDistance() throws ParsingException {
+        Properties bindings = createBindings();
+        String a = "aaaa";
+        String b = "bbb";
+        // special values used in custom clustering via UserDefinedDistance
+        bindings.put("value1", a);
+        bindings.put("value2", b);
+
+        long expectedLength = a.length() - b.length();
+
+        Evaluable evaluable = (new ClojureParser()).parse("(- (count value1) (count value2))", "clojure");
+        Object result = evaluable.evaluate(bindings);
+        assertEquals(result, expectedLength, "Length difference should be the same.");
+    }
+}