Repository: astropy/pyvo

Issue:
AsyncTAPJob's "query" attribute is not robust to case variation among community TAP servers
We have recently noticed that the "query" attribute of AsyncTAPJob (https://pyvo.readthedocs.io/en/latest/api/pyvo.dal.AsyncTAPJob.html#pyvo.dal.AsyncTAPJob.query) does not work with either the Rubin Observatory TAP service or the CADC TAP service under some conditions.

"query" works by interrogating the `<uws:parameters>` element of the UWS job's XML for a `<uws:parameter>` whose "id" attribute is "query" - that is, it's trying to find the value of the URL parameter "QUERY" from the original TAP invocation, which UWS says should be reflected back in this XML.

The problem here stems from the fact that, while the IVOA standards, including TAP, generally explicitly document the names of the services' URL query parameters in UPPERCASE, the DALI standard makes clear that _in the URL interface, a conformant service must accept any case rendering of the parameter name_.  So "query", "QUERY", and "Query", for instance, are equally valid.

The UWS standard does not appear to say anything about how query parameters should be reflected back - in their original case, or in a standardized case.  It appears that some of the well-known community TAP services return them in their original case, while others force them to lowercase.

Notably, the OpenCADC TAP service seems to return parameters in their original case, which seems like a reasonable thing to do in the absence of words to the contrary in UWS.  The Rubin Observatory also uses this code base, so our TAP service is affected by the same issue.

There is existing community code that generates queries with the parameter names in uppercase, following the style of all the IVOA documentation (though, of course, this is not required) and we are currently unable to work successfully with the resulting UWS jobs.

It seems as if PyVO should apply the same case-blindness here that DALI requires to be used on input, and recognize any case form of "QUERY" in its implementation.

The Rubin team is willing to make this change but wanted to trigger a discussion first to see if there will be resistance to this.

Potentially interested parties: @rra @athornton @pdowler @cbanek 

Comments:

Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index 5b9953d..c6ada61 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -19,6 +19,9 @@
 
 - Allow session to be passed through in SSA and DataLink. [#327]
 
+- pyvo.dal.tap.AsyncTAPJob treats parameter names as case-insensitive when
+  retrieving the query from the job record. [#356]
+
 
 1.3 (2022-02-19)
 ==================
diff --git a/pyvo/dal/tap.py b/pyvo/dal/tap.py
index c414872..4a7043e 100644
--- a/pyvo/dal/tap.py
+++ b/pyvo/dal/tap.py
@@ -646,7 +646,7 @@ class AsyncTAPJob:
         """
         self._update()
         for parameter in self._job.parameters:
-            if parameter.id_ == 'query':
+            if parameter.id_.lower() == 'query':
                 return parameter.content
         return ''
 


Test patch:
diff --git a/pyvo/dal/tests/test_tap.py b/pyvo/dal/tests/test_tap.py
index e06ec6f..d419638 100644
--- a/pyvo/dal/tests/test_tap.py
+++ b/pyvo/dal/tests/test_tap.py
@@ -12,7 +12,7 @@ from urllib.parse import parse_qsl
 import pytest
 import requests_mock
 
-from pyvo.dal.tap import escape, search, TAPService
+from pyvo.dal.tap import escape, search, AsyncTAPJob, TAPService
 from pyvo.io.uws import JobFile
 from pyvo.io.uws.tree import Parameter, Result
 
@@ -105,7 +105,7 @@ class MockAsyncTAPServer:
         job.destruction = Time.now() + TimeDelta(3600, format='sec')
 
         for key, value in data.items():
-            param = Parameter(id=key.lower())
+            param = Parameter(id=key)
             param.content = value
             job.parameters.append(param)
 
@@ -175,11 +175,11 @@ class MockAsyncTAPServer:
             if 'QUERY' in data:
                 assert data['QUERY'] == 'SELECT TOP 42 * FROM ivoa.obsCore'
                 for param in job.parameters:
-                    if param.id_ == 'query':
+                    if param.id_.lower() == 'query':
                         param.content = data['QUERY']
             if 'UPLOAD' in data:
                 for param in job.parameters:
-                    if param.id_ == 'upload':
+                    if param.id_.lower() == 'upload':
                         uploads1 = {data[0]: data[1] for data in [
                             data.split(',') for data
                             in data['UPLOAD'].split(';')
@@ -426,19 +426,46 @@ class TestTAPService:
     @pytest.mark.usefixtures('async_fixture')
     def test_submit_job(self):
         service = TAPService('http://example.com/tap')
-        job = service.submit_job(
-            'http://example.com/tap', "SELECT * FROM ivoa.obscore")
+        job = service.submit_job("SELECT * FROM ivoa.obscore")
 
         assert job.url == 'http://example.com/tap/async/' + job.job_id
         assert job.phase == 'PENDING'
         assert job.execution_duration == TimeDelta(3600, format='sec')
         assert isinstance(job.destruction, Time)
         assert isinstance(job.quote, Time)
+        assert job.query == "SELECT * FROM ivoa.obscore"
 
         job.run()
         job.wait()
         job.delete()
 
+    @pytest.mark.usefixtures('async_fixture')
+    def test_submit_job_case(self):
+        """Test using mixed case in the QUERY parameter to a job.
+
+        DALI requires that query parameter names be case-insensitive, and
+        some TAP servers reflect the input case into the job record, so the
+        TAP client has to be prepared for any case for the QUERY parameter
+        name.
+        """
+        service = TAPService('http://example.com/tap')
+
+        # This has to be tested manually, bypassing the normal client layer,
+        # in order to force a mixed-case parameter name.
+        response = service._session.post(
+            "http://example.com/tap/async",
+            data={
+                "REQUEST": "doQuery",
+                "LANG": "ADQL",
+                "quERy": "SELECT * FROM ivoa.obscore",
+            }
+        )
+        response.raw.read = partial(response.raw.read, decode_content=True)
+        job = AsyncTAPJob(response.url, session=service._session)
+
+        assert job.url == 'http://example.com/tap/async/' + job.job_id
+        assert job.query == "SELECT * FROM ivoa.obscore"
+
     @pytest.mark.usefixtures('async_fixture')
     def test_modify_job(self):
         service = TAPService('http://example.com/tap')