Repository: opendatacube/datacube-core

Issue:
New "datacube dataset search" spatial extent parameters not working
Recently updated to ODC v1.7.0 and adjusting scripts for new way of searching for time & spatial extent etc, changing from "lower val < param < upper val" to param in [lower val, upper val] as directed using the help section of "datacube dataset search". This works for querying by a time search, but it is trying to convert to a datetime for spatial extent searches.

Running a search for: 
`datacube dataset search product=s2_netCDF_20m "lon in [109,109.5]"`

returns the following error message:
```
Traceback (most recent call last):
  File "/opt/miniconda3/envs/mdc1_7/lib/python3.6/site-packages/pandas/core/arrays/datetimes.py", line 1861, in objects_to_datetime64ns
    values, tz_parsed = conversion.datetime_to_datetime64(data)
  File "pandas/_libs/tslibs/conversion.pyx", line 200, in pandas._libs.tslibs.conversion.datetime_to_datetime64
TypeError: Unrecognized value type: <class 'str'>

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/miniconda3/envs/mdc1_7/bin/datacube", line 10, in <module>
    sys.exit(cli())
  File "/opt/miniconda3/envs/mdc1_7/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/opt/miniconda3/envs/mdc1_7/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/opt/miniconda3/envs/mdc1_7/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/miniconda3/envs/mdc1_7/lib/python3.6/site-packages/click/core.py", line 1135, in invoke
    sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)
  File "/opt/miniconda3/envs/mdc1_7/lib/python3.6/site-packages/click/core.py", line 641, in make_context
    self.parse_args(ctx, args)
  File "/opt/miniconda3/envs/mdc1_7/lib/python3.6/site-packages/click/core.py", line 940, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/opt/miniconda3/envs/mdc1_7/lib/python3.6/site-packages/click/core.py", line 1477, in handle_parse_result
    self.callback, ctx, self, value)
  File "/opt/miniconda3/envs/mdc1_7/lib/python3.6/site-packages/click/core.py", line 96, in invoke_param_callback
    return callback(ctx, param, value)
  File "/opt/miniconda3/envs/mdc1_7/lib/python3.6/site-packages/datacube/ui/click.py", line 349, in my_parse
    parsed_expressions = parse_expressions(*list(value))
  File "/opt/miniconda3/envs/mdc1_7/lib/python3.6/site-packages/datacube/ui/expression.py", line 326, in parse_expressions
    out.update(expr.as_query())
  File "/opt/miniconda3/envs/mdc1_7/lib/python3.6/site-packages/datacube/ui/expression.py", line 298, in as_query
    return {self.field_name: _time_to_search_dims([self.low_value.value, self.high_value.value])}
  File "/opt/miniconda3/envs/mdc1_7/lib/python3.6/site-packages/datacube/api/query.py", line 289, in _time_to_search_dims
    start = _to_datetime(tr_start)
  File "/opt/miniconda3/envs/mdc1_7/lib/python3.6/site-packages/datacube/api/query.py", line 268, in _to_datetime
    return pandas_to_datetime(t, utc=True, infer_datetime_format=True).to_pydatetime()
  File "/opt/miniconda3/envs/mdc1_7/lib/python3.6/site-packages/pandas/core/tools/datetimes.py", line 611, in to_datetime
    result = convert_listlike(np.array([arg]), box, format)[0]
  File "/opt/miniconda3/envs/mdc1_7/lib/python3.6/site-packages/pandas/core/tools/datetimes.py", line 302, in _convert_listlike_datetimes
    allow_object=True)
  File "/opt/miniconda3/envs/mdc1_7/lib/python3.6/site-packages/pandas/core/arrays/datetimes.py", line 1866, in objects_to_datetime64ns
    raise e
  File "/opt/miniconda3/envs/mdc1_7/lib/python3.6/site-packages/pandas/core/arrays/datetimes.py", line 1857, in objects_to_datetime64ns
    require_iso8601=require_iso8601
  File "pandas/_libs/tslib.pyx", line 460, in pandas._libs.tslib.array_to_datetime
  File "pandas/_libs/tslib.pyx", line 685, in pandas._libs.tslib.array_to_datetime
  File "pandas/_libs/tslib.pyx", line 809, in pandas._libs.tslib.array_to_datetime_object
  File "pandas/_libs/tslib.pyx", line 803, in pandas._libs.tslib.array_to_datetime_object
  File "pandas/_libs/tslibs/parsing.pyx", line 82, in pandas._libs.tslibs.parsing.parse_datetime_string
ValueError: Given date string not likely a datetime.
```
This is using ODC v1.7.0 on Ubuntu 18.04. Pandas version is 0.24.2, click version is 7.0.
Using the old style of search still works but with a depreciation warning. Also tried using "x" instead of lon and "y" instead of lat but got the same result.



Discussion:
@uchchwhash looks like in this change https://github.com/opendatacube/datacube-core/commit/ce9c4c9ae2cca5b2df625cb85ab0cd0defcdcb2e you are assuming that "between expression" contains dates only, but it should also work for numeric types like `lon|lat|x|y`, any reason timestamp parsing needs to happen at that level, why not keep behaviour of the previous version of `BetweenExpression`?

@JackLidge thanks for reporting, work around for now is to keep using old style expression and ignore warnings. I'm curious what your overall workflow is like? I didn't think command line searching was used much.



Resolving patch:
diff --git a/datacube/ui/expression.py b/datacube/ui/expression.py
--- a/datacube/ui/expression.py
+++ b/datacube/ui/expression.py
@@ -295,7 +295,10 @@ def query_repr(self, get_field):
         return get_field(self.field_name).between(search_range.begin, search_range.end)
 
     def as_query(self):
-        return {self.field_name: _time_to_search_dims([self.low_value.value, self.high_value.value])}
+        if self.field_name == 'time':
+            return {self.field_name: _time_to_search_dims([self.low_value.value, self.high_value.value])}
+        else:
+            return {self.field_name: Range(self.low_value.as_value(), self.high_value.as_value())}
 
 
 class ExpressionList(List):


Test patch:
diff --git a/tests/ui/test_expression_parsing.py b/tests/ui/test_expression_parsing.py
new file mode 100644
--- /dev/null
+++ b/tests/ui/test_expression_parsing.py
@@ -0,0 +1,21 @@
+from datacube.ui import parse_expressions
+from datacube.model import Range
+from datetime import datetime
+
+
+def test_between_expression():
+    q = parse_expressions('time in [2014, 2015]')
+    assert 'time' in q
+    r = q['time']
+    assert isinstance(r, Range)
+    assert isinstance(r.begin, datetime)
+    assert isinstance(r.end, datetime)
+
+    for k in ('lon', 'lat', 'x', 'y'):
+        q = parse_expressions('{} in [10, 11.3]'.format(k))
+        assert k in q
+        r = q[k]
+        assert isinstance(r, Range)
+        assert isinstance(r.begin, float)
+        assert isinstance(r.end, float)
+        assert r == Range(10, 11.3)